diff --git a/.gitignore b/.gitignore index 0a11ee3..7939cf2 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,5 @@ -# fill later +# Private, composed deployment configuration. Install into /etc/orchestra/ +# only after all environment-specific values have been filled in. +.orchestra-config/ +clients/ +/orchestra diff --git a/AUDIT.md b/AUDIT.md new file mode 100644 index 0000000..6d7fd89 --- /dev/null +++ b/AUDIT.md @@ -0,0 +1,475 @@ +# Orchestra — spec conformance audit & remediation plan + +Audited 2026-07-27 against `orchestra-spec (1).md` at commit `325c684`. +Method: read every non-test file in `internal/` and `cmd/`, traced each spec +section to its call site, and checked whether the live path (`main.go` → +router → coordinator → adapter) actually reaches it. + +--- + +## Verdict + +The substrate (Layer 1) is genuinely built. Layers 2–4 are **shaped** but not +**wired**: the packages exist, have tests, and compile — but the code paths +that would make an unattended run work are either dead, mis-keyed, or built +against herdr methods that appear to be invented. + +Concretely: **a task can be created, routed, and leased. It can never +complete, and it can never rotate.** After 3 lease expiries the router marks +it `TaskFailed`. That is the whole of "connection works but features are +underdeveloped." + +`progress.md` overstates completion in several places (see §Corrections). + +| Spec layer | State | +|---|---| +| L1 substrate (§3, §4) | Built and correct in the main path. Some defects (below), no missing mechanism. | +| L2 harness (§5) | **Not functional.** Occupancy broken, rotation unreachable, adapter protocol unverified. | +| L3 continuity (§6) | **Dead code.** Handoff schema, pickup validation, scratch branches, TASK.md — none reached from the live path. | +| L4 surfaces (§7) | Partial. Brief/standup/delivery exist; quota projection has no producer; authz has a bypass. | + +--- + +## Blocking defects — in dependency order + +### B1. Occupancy is measured from the wrong value (§5.2.1) + +`herdr/adapter.go:191` + +```go +u, e := a.Usage(s.PaneID) // ClaudeUsage(path string) wants a transcript file +``` + +`ClaudeUsage`/`CodexUsage`/`OpenCodeUsage` all take a **filesystem path** to +session state. They are handed a **herdr pane ID**. Every call returns +`open : no such file`, so `Coordinator.rotate` hits +`if err != nil { continue }` on line 430 and never rotates anything. + +The spec is unusually explicit here — §5.2.1: *"Build this measurement and +verify it against a live session **before** wiring any trigger — the whole +rotation system rests on this number."* That step was skipped, and the +verification gap is exactly where the bug is. + +Compounding: `CodexActiveUsage` (the discovery function that resolves the +active rollout via `state_*.sqlite`) and `ClaudeStopHookUsage` and +`OpenCodeStatus` are written but **called from nowhere** outside tests. + +### B2. The adapter lookup key is wrong in three of four call sites (§5.3, §5.4) + +`AdapterFactory.Herdrs` is keyed by **herdr instance id** (`homesrv-claude`). +`Session.Harness` is set by `CLIAdapter.Lease` to the **harness kind** +(`claude`). Then: + +| Call site | Key used | Resolves? | +|---|---|---| +| `refreshSessionHealth` (orchestrator.go:221) | `HerdrID`, falls back to lease | yes | +| `Reconcile` (:291) | `session.Harness` | **no** | +| `expire` (:376, :397) | `session.Harness` | **no** | +| `rotate` (:419) | `session.Harness` | **no** | + +So: orphaned panes are never killed on restart, `pane.exited` fast-path never +fires, expired leases never kill their pane, and rotation exits before it +begins. All silently — every one is a bare `continue`. + +### B3. Nothing emits `TaskCompleted` (§4, §5.2) + +Grep for producers: there is exactly one, `POST /v1/tasks/{id}/complete`, +which a human has to call. The spec assigns this to the **stop-hook/wrapper** +("the plane emits events, not the agent" — Invariant 2). There is no stop +hook, no wrapper, no completion detection of any kind. + +Consequence: an agent that finishes its work sits idle until the 30-minute +lease TTL expires → `TaskReleased` → re-leased → repeat → `TaskFailed` at +attempt 3. Unattended runtime, the one metric in §0, is currently bounded at +90 minutes per task and always ends in a false failure. + +### B4. The router counts rotation as a retry (§5.3, §5.4) + +`router.go:127` increments `attempts[TaskID]` on **every** `TaskReleased`, +and `:169` increments again on every lease. `:149` fails the task at +`MaxAttempts` (3, from `main.go:64`). + +Rotation *is* `TaskReleased` (§5.3: "rotation = intra-task lease transfer"). +So a task healthy enough to rotate twice is killed by the retry limit. The +spec's retry policy is for *failures* (§5.4), not lease transfers. These need +separate counters — `attempts` should only advance on expiry/crash releases, +never on a release carrying a valid `handoff_ref`. + +### B5. Herdr protocol methods are unverified and at least partly invented + +`adapter.go` calls `pane.release`, `pane.kill`, `pane.rotation_signal`, +`pane.read`, `agent.get`, `agent.start`, `worktree.create/open`, +`agent.prompt`. The file itself documents that a previous method +(`pane.status`) *was not a valid protocol method* — so this surface has a +history of being written against an imagined API. + +`pane.rotation_signal` is near-certainly not real: herdr is a generic +multiplexer with no concept of Orchestra rotation. And `pane.release` +returning a `handoff_ref` is architecturally wrong regardless of whether the +method exists — **herdr does not write handoffs; the agent does** (§6.1, the +handoff is a CAS artifact produced under the Face-B stop hook). + +Nothing here can be settled from source. It needs `herdr api schema --json` +run against your protocol-17 instance, diffed against every method the +adapter calls. + +Related: §5.1 requires `agent.prompt` with **inline `wait`** for bootstrap +injection, because that's what closes the send-into-a-half-rendered-prompt +race. `CLIAdapter.Lease` calls `Prompt(..., wait=0)` — no wait. Only +`Bootstrap` passes one. + +### B6. Layer 3 is entirely dead code (§6) + +Never called from anything but tests: +`continuity.ValidatePickup`, `ScratchCommit`, `ScratchPush`, `ScratchPull`, +`ScratchSync`, `MarkdownChanges`, `Save`, `Load`, `Encode`, `Decode`. + +`VerifyTaskFile` *is* called — but only when `GitWorktrees.TaskFileSHA` is +non-empty, and `main.go:112` constructs the worktrees **without ever setting +it**. Nothing writes a `TASK.md` into a worktree in the first place. + +So the entire §6.2 pickup contract — the part the spec calls +"the make-or-break" and "the defense against the telephone game" — does not +execute. Rotation, if B1/B2 were fixed, would hand the next agent a +`handoff_ref` that was never schema-validated, never anchor-checked, and +never accompanied by an immutable spec. + +### B7. Quota projection has no producer (§7.2) + +`QuotaAvailability.sumSince` folds `QuotaReported` events. `AggregateQuota` +does the same for the brief. **No code appends a `QuotaReported` event.** +Your config sets `quota_limit_5h: 50` / `quota_limit_weekly: 500` on all six +herdrs; those limits are compared against a permanent 0. The availability +filter is a no-op and the brief's `quota_consumed` is always `{}`. + +### B8. `Surface: system` is an unauthenticated full-control bypass (§7.1) + +`authz.CapabilityFor` grants `System` → `FullControl`. `authz.HTTP` reads the +surface straight from the `X-Orchestra-Surface` header, and the token map in +`main.go:760` has **no entry for `system`** — so `tokens[System]` is `""` and +the token check is skipped entirely. Any request on the LAN with +`X-Orchestra-Surface: system` can emit any event on any task. + +The surface is also absent from `ParseSurface`'s intent: `System` is supposed +to mean "the plane itself, in-process", but it's reachable over HTTP. It +should never be accepted from a request header. + +(Also: unset `ORCHESTRA_*_TOKEN` means that surface is unauthenticated; your +`.orchestra-config/orchestra.env` sets none of them. Defensible on WireGuard, +worth a deliberate decision rather than an accident.) + +--- + +## Secondary defects + +| # | Where | Issue | +|---|---|---| +| S1 | `operations.go:16,25` | `go vet` fails: `Brief.From/To` both serialize as `"from"`; `GitSync.Branch/Head/Status` all as `"branch"`. The brief's git state is unreadable by any client. | +| S2 | `main.go:238` | Brief's git state is read from `ORCHESTRA_DATA` (the event-log dir), not from the project worktrees. §7.4 wants "what pushed, what's on which branch, what workpc still needs to pull" — per project. | +| S3 | `operations.BuildBrief` | Counts completions but never surfaces `report_ref`/`receipt`, which §7.4 names as the proofs the brief exists to carry. | +| S4 | `delivery.Fanout.Run:110` | A single send error `return`s and permanently kills the notification goroutine — silently, since `main.go:729` only logs. One ntfy hiccup at 2am = no notifications for the rest of the night. Cursor is also in-memory, so a restart re-notifies the entire log from seq 0. | +| S5 | `store.Lease:341`, `ExpireLeases:350` | `Event.ID` is set to the **task id**, so event IDs collide across every lease of a task. `ApplyAdvisory` and any future ID-based lookup are unsound. | +| S6 | `store.Append:191` | Ingest dedup returns `nil` (success) without appending; `main.go:175` then returns `s.Events(0)[len-1]` — an unrelated event — with `201 Created`. | +| S7 | `store.apply:154` | `TaskAmended` applies only `title`. §4 lists title, due, description, `inherent_priority`. Amendments to the others are accepted, logged, and silently ignored by the projection. | +| S8 | §3.1 | No compensation-event mechanism exists. Append-only holds, but the spec's *correction* path ("a compensating event is appended") has no implementation. | +| S9 | `main.go:482` | Two independent lease-expiry loops (`main.go` 1s ticker and `Coordinator.expire` 30s) race on the same reclaim. Harmless today only because version CAS rejects the loser. | +| S10 | federation | `federation.Registry.Register` accepts a self-declared `id` + self-chosen `token` from any caller — registration is admission-control-free. | +| S11 | §5.3 | Thrash detection, the soft ~55% threshold, milestone rotation, and agent-initiated `ROTATE` are all absent. Only the hard threshold exists, and it's unreachable (B1). | + +--- + +## The architectural fork — decided 2026-07-27 + +There are **two incompatible federation designs** in the tree, and only one is +deployed. + +### Design A — "drive the remote socket" (deployed) + +`clients/herdr-bridge.go` runs on workpc and proxies its local herdr Unix +socket to `192.168.1.105:9245`. `.orchestra-config/config.jsonc` registers +`workpc-claude` / `workpc-codex` / `workpc-opencode` against that machine, so +the **homesrv** coordinator resolves a workpc herdr as a candidate and calls +`worktree.create`, `agent.start`, `agent.prompt` on it over TCP as if it were +local. workpc is a dumb pane host; all orchestration logic and all state live +on homesrv, and live state crosses the machine boundary continuously. + +### Design B — "workers pull tasks" (`/v1/federation/*`, unused) + +workpc would run its own Orchestra process that registers itself +(`POST /v1/federation/workers`), heartbeats, polls the event log with a cursor +(`GET /v1/federation/events` + `/ack`), claims its own lease +(`POST .../claim`), runs a **local** coordinator against its **local** herdr +socket, and reports a handoff carrying an `anchor_sha` it computed from its +own checkout (`POST .../handoff`). homesrv stays authoritative for the log; +nothing but git and validated artifacts crosses the wire. + +This is what the spec describes — §2.1: *"Everything crossing a machine +boundary is git + a validated artifact — never live state over the wire."* +**It is fully written on the server side and has zero clients.** There is no +worker binary in this repo; every one of those endpoints is unreachable in +the current deployment. + +### Why keeping both is harmful, not merely untidy + +Design A breaks a specific correctness property Design B was written to hold. + +**The anchor is validated on the wrong machine.** `Coordinator.rotate` +(orchestrator.go:460) calls `herdr.HeadSHA(session.Worktree)`, which shells +out to `git -C rev-parse HEAD`. The coordinator runs on homesrv; +`session.Worktree` is a path on **workpc**. Two outcomes, both bad: + +- the path doesn't exist on homesrv → `HeadSHA` errors → rotation silently + skips forever (another bare `continue`); or +- the path *does* exist on homesrv — likely, since every project shares the + `/var/lib/orchestra/worktrees//` layout — → it returns + **homesrv's HEAD for an unrelated checkout**, and the emitted `TaskReleased` + certifies a commit the agent never worked on. + +The second is the dangerous one: it passes validation, looks correct, and +hands the successor an anchor describing another repo's state. That is exactly +the failure §9 item 8 names — *"worktree/anchor validation when a lease's git +checkout lives on a different host than the router"* — against the +degrade-safe default the spec already supplies: *"validate against the local +checkout wherever the harness runs."* Design B satisfies this by construction; +Design A cannot satisfy it without moving validation out of the coordinator. + +The same class of bug applies to per-host quota accounting (§9 item 8) and to +`cleanupCompleted`, which runs `git worktree remove` on homesrv for a worktree +that lives on workpc. + +### Latent defects in the unused half + +Never surfaced because nothing exercises them: + +- **Offline detection only runs inside `Snapshot()`** (federation.go:103-118), + called solely from `GET /v1/federation/workers`. Nothing polls it, so + `OnOffline` — the hook at `main.go:131` that releases leases held by a + vanished worker — fires only if a human hits that endpoint. The TTL backstop + §5.4 relies on does not tick on its own. +- **The registry is in-memory with no persistence.** Every restart forgets all + workers and their cursors; a reconnecting worker re-reads the entire log + from seq 0. + +### Decision + +**Keep Design A through Phase 5; commit to Design B in Phase 6.** Phases 1–4 +(occupancy, Face B, rotation, continuity) are single-host concerns, provable +on homesrv alone with workpc's herdr as just another pane host. Resolving the +federation question first would block the fixes that actually unblock +unattended runtime. + +Two guardrails land **immediately**, so Design A cannot corrupt state in the +meantime: + +1. Anchor computation happens where the checkout is. Under the bridge that + means having herdr run the `rev-parse` in the pane's worktree rather than + shelling out locally — or, if the protocol schema won't support it, + **refusing to rotate any lease held by a non-local herdr** until Phase 6. A + loud refusal beats a false anchor. +2. Same treatment for `cleanupCompleted`'s `git worktree remove`. + +Phase 6 then builds the worker binary against the endpoints that already +exist, adds server-issued tokens (S10), persists the registry, and drives +offline detection from a ticker rather than a request handler. + +What is explicitly **not** acceptable is leaving both designs in place +unmarked. If the worker binary is ever abandoned, delete `/v1/federation/*` +and record the deviation — as it stands the repo reads as though the +spec-conformant path is implemented when nothing can reach it. + +--- + +## Remediation plan + +Ordered so each phase is verifiable on its own and nothing depends on an +unproven layer below it — the spec's own build discipline (§8). + +### Phase 0 — Ground truth (half a day, blocks everything) + +Nothing below is safe to write until the herdr surface is known. + +1. On workpc: `herdr api schema --json > herdr-schema.json`, commit it to + `deploy/`. +2. Write `internal/herdr/schema_test.go`: for every method string in + `adapter.go`, assert it exists in the committed schema with the params we + send. This is the test that would have caught `pane.status` and will catch + the next one. +3. Manually drive one pane end to end against real herdr — create worktree, + start agent, prompt with inline wait, read status, kill — and record the + actual request/response shapes. Fix `adapter.go` to match. +4. Delete `pane.rotation_signal` and the `RotationSignal` interface unless the + schema proves it exists. + +**Done when:** `schema_test.go` passes against the committed schema, and a +scripted manual run starts and stops a real Claude Code pane on workpc. + +### Phase 1 — Occupancy, measured for real (§5.2.1) — fixes B1 + +The spec says build this first and verify against a live session. Do exactly +that, and nothing else in this phase. + +1. Add `SessionFile string` to `herdr.Session`. Resolve it at lease time, per + harness: + - **claude** — the transcript path. Cleanest source is the Stop hook's + stdin (`transcript_path`), which you need for Phase 2 anyway; until then + resolve `~/.claude/projects//.jsonl` by + newest-mtime under the encoded worktree dir. + - **codex** — `CodexActiveUsage` already does the `state_*.sqlite → + threads.rollout_path` discovery. Call it. Filter by the rollout whose cwd + matches the worktree. + - **opencode** — `OpenCodeStatus` against `:4096` as fast path, + `~/.local/share/opencode/storage/message/` as backstop, per §5.2.1's + "the SSE stream is not rock-solid". +2. Change `CLIAdapter.Occupancy` to use `s.SessionFile`, and make a missing + session file a **hard error surfaced in `MonitorHealth`**, never a silent + `continue`. +3. Keep the §5.2.1 trap explicit: assert in a test that a fixture with a large + cumulative total but small last-turn usage yields *low* occupancy. This is + the failure mode the spec singles out. +4. Add `GET /v1/tasks/{id}/occupancy` returning the raw numerator, the window, + and the fraction — so you can eyeball it against a live session before + trusting it to drive rotation. + +**Done when:** a real Claude Code session at a known context fill reports a +fraction within a few points of what `/context` says. + +### Phase 2 — Face B and completion (§5.2, §4) — fixes B3, B5 + +1. **Claude Code Stop hook** (`deploy/hooks/orchestra-stop.sh`, bash, no + deps — §5.6). Reads hook JSON on stdin, POSTs `transcript_path` + task id + to a new `POST /v1/harness/turn` endpoint. Exit 2 with stderr when the + plane says "rotate but no valid handoff yet" — that's the spec's refuse-the- + turn mechanism, and `ClaudeStopHookUsage` already parses the input shape. +2. `POST /v1/harness/turn` computes occupancy, evaluates the rotation triggers, + and returns a decision: `continue` | `prepare_handoff` (soft) | + `rotate_now` (hard) | `refuse` (over threshold, handoff missing/invalid). +3. **Completion**: `POST /v1/harness/complete` — the wrapper/hook path that + uploads the report to CAS and emits `TaskCompleted` with `report_ref` + + `receipt`. Wire the receipt from Phase 1's occupancy reader, **summed + across lease intervals** per §4. +4. Codex/opencode Face B: rollout-tail and SSE `session.status` respectively, + polled by the coordinator rather than hook-pushed. Same decision endpoint. + +**Done when:** a task given a trivial goal on a real harness reaches +`TaskCompleted` with a `report_ref` resolvable from CAS, with no human action. + +### Phase 3 — Make rotation reachable and correct (§5.3, §5.4) — fixes B2, B4 + +1. Key sessions by `HerdrID` everywhere. Add a single + `Coordinator.adapterFor(session) (herdr.Adapter, error)` and route all four + call sites through it. Add a regression test that registers an adapter + under `homesrv-claude`, leases with `Harness: "claude"`, and asserts + rotation still fires. +2. Replace every silent `continue` in `rotate`/`expire` with a recorded reason + on `MonitorHealth`. The class of bug in B1/B2 is only invisible because of + those bare continues. +3. Split the router's counters: `releases` (informational) vs `failures` + (drives `MaxAttempts`). A `TaskReleased` carrying a valid `handoff_ref` + must not advance `failures`. Test: a task that rotates 5 times is still + alive. +4. Implement the missing triggers (§5.3): soft 55% → `prepare_handoff`; + milestone; thrash (N failed test runs / same file M times / identical tool + calls) as a circuit breaker with `reason=thrash` and populated `dead_ends`; + agent-initiated `ROTATE`. +5. Make TTL and the retry policy config, not the hardcoded `30*time.Minute` at + `router.go:165` and `1800` in `main.go` (§5.4: "N, ttl, retry backoff = + config"). +6. **Land the two Design-A guardrails** (see the federation decision above): + compute the anchor where the checkout is, or refuse to rotate a lease held + by a non-local herdr; same for `cleanupCompleted`'s worktree removal. These + are the fixes that make it safe to defer federation to Phase 6. + +**Done when:** a long-running task crosses 75% occupancy, rotates at a turn +boundary, and the successor continues — twice in a row, unattended. + +### Phase 4 — Wire Layer 3 into the live path (§6) — fixes B6 + +This is where the dead code becomes load-bearing. + +1. **Write `TASK.md`** into the worktree at creation from the `TaskCreated` + payload; hash it; store the hash on the task; pass it as `TaskFileSHA` into + `PerProjectGitWorktrees` (`main.go:112`). Re-inject it on every rotation + bootstrap — §6.2 is explicit that this is what defeats the telephone game. +2. **Handoff production**: the rotating agent writes the §6.1 TOML/JSON + handoff; the stop hook uploads it via `POST /v1/artifacts`; the plane + `continuity.Decode`s it, rejecting free-form `[knowledge]` (Invariant 4), + and only then emits `TaskReleased` with the hash. Never accept a + `handoff_ref` the plane hasn't validated. +3. **Scratch-branch commit before release** (`ScratchCommit`), so §6.2 step 3 + collapses to one sha compare. `ScratchCommit` already refuses to commit a + dirty `TASK.md` — good, keep that. +4. **Pickup**: `ValidatePickup` runs in `Coordinator.Start` before the + successor's bootstrap prompt. Fail → do not start; emit `TaskBlocked`. +5. Replace `CLIAdapter.Bootstrap`'s freeform prose with the §6.2 procedure + (~200 tokens: read handoff, run validate-handoff, re-read TASK.md, + proceed), and use `agent.prompt`'s inline wait. +6. Wire `MarkdownChanges` to the §6.3 adjacent-task notice, or delete it and + record the deferral. Dead code that looks implemented is what produced this + audit. + +**Done when:** a rotation whose anchor has drifted is *refused*, visibly, and +one whose anchor is clean proceeds without the successor re-deriving context. + +### Phase 5 — Close the surfaces (§7) — fixes B7, B8, S1–S4 + +1. Emit `QuotaReported`. Source: the same per-harness session state as + Phase 1, on a timer per `harness+window` (§7.2 — "a projection keyed by + `harness+window`, fed by the same per-harness session state"). Until then, + your quota limits are decorative. +2. Reject `X-Orchestra-Surface: system` at the HTTP boundary unconditionally. + `System` must be constructible in-process only. Then decide explicitly + whether the remaining surfaces require tokens on your LAN, and set them. +3. Fix the `go vet` json-tag collisions (S1) — the brief is currently + unparseable for git state. +4. Brief: per-project git sync state from the actual worktrees (S2), and + include `report_ref`/`receipt` in the rollup (S3). +5. Delivery: never `return` on send error — log, back off, continue. Persist + the cursor next to the event log (S4). Deliver the morning brief itself. + +**Done when:** `/v1/brief` for an overnight window shows real quota per +harness, real per-project git state, and the receipts for every completion. + +### Phase 6 — Federation, decided (§2.1, §9 item 8) — fixes the fork, S10 + +Only after Phases 1–5 run clean on a single host. + +1. Anchor validation and `HeadSHA` execute **where the checkout is**. Today + they run on homesrv against a workpc path. +2. Either build the worker binary against `/v1/federation/*` (git-only + transport, per spec) or delete that API and document the bridge as a + deliberate deviation. Do not keep both. +3. Add admission control to worker registration (S10): server-issued tokens, + not self-declared. +4. Then run the real two-machine overnight batch that §9 item 8 asks for. + +--- + +## Corrections to `progress.md` + +Worth fixing, because the next session will otherwise trust it: + +- "Continuity: strict handoff schema/validation, CAS save/load, pickup + validation, scratch-branch commit/push/pull helpers" — accurate as a + description of the *package*, misleading as a description of the *system*. + None of it is reachable at runtime. +- "herdr adapters with native occupancy readers" — the readers exist; they are + called with the wrong argument and always fail. +- "Router: ... quota availability at conservative 80% threshold" — the + threshold logic is correct, but no `QuotaReported` event is ever produced, + so it evaluates against zero. +- The rotation fix described in "Verified fixed this pass" is real and + correct — but `rotate` cannot reach that code, because the adapter lookup + above it (B2) fails first. The test passes because it registers the adapter + under the harness kind rather than the herdr id, which the production + config never does. +- "`go build ./...` and `go test ./...` both pass" — true; `go vet ./...` + does not. + +--- + +## Suggested order of attack + +If you want one thing to do today: **Phase 0**, then **B2** (a ~20-line fix +that makes three subsystems reachable), then **B1**. Those three turn a system +that cannot rotate into one that can, and everything else in the plan is +building on top rather than repairing underneath. diff --git a/cmd/orchestra/main.go b/cmd/orchestra/main.go index d84a8ab..e1d0831 100644 --- a/cmd/orchestra/main.go +++ b/cmd/orchestra/main.go @@ -5,6 +5,7 @@ import ( "encoding/json" "io" "log" + "net" "net/http" "orchestra/internal/admin" "orchestra/internal/authz" @@ -26,6 +27,24 @@ import ( ) func id() string { return domain.NewID() } + +const defaultHerdrPort = "9245" + +func herdrAddress(rr registry.Registry, h registry.Herdr) string { + if h.Address != "" { + return h.Address + } + m, ok := rr.Machine(h.MachineID) + if !ok { + return "" + } + host, _, err := net.SplitHostPort(m.Address) + if err != nil { + return m.Address + } + return net.JoinHostPort(host, defaultHerdrPort) +} + func main() { dir := os.Getenv("ORCHESTRA_DATA") if dir == "" { @@ -37,27 +56,34 @@ func main() { } var rr registry.Registry var rt *router.Router + var coordinator *orchestrator.Coordinator if config := os.Getenv("ORCHESTRA_CONFIG"); config != "" { if rr, err = registry.Load(config); err != nil { log.Fatalf("load orchestra config: %v", err) } rt = &router.Router{Store: s, Registry: rr, Reachability: registry.TCPReachability{}, Timeout: time.Second, Retry: router.RetryPolicy{MaxAttempts: 3, Backoff: time.Minute}} - limits := map[string]float64{} + limits := map[string]router.QuotaWindowLimits{} for _, h := range rr.Herdrs() { - if h.QuotaLimit > 0 { - limits[h.ID] = h.QuotaLimit + w := router.QuotaWindowLimits{FiveHour: h.QuotaLimit5h, Weekly: h.QuotaLimitWeekly} + if w.Weekly <= 0 && h.QuotaLimit > 0 { + // Back-compat: the old single-window field meant weekly. + w.Weekly = h.QuotaLimit + } + if w.FiveHour > 0 || w.Weekly > 0 { + limits[h.ID] = w } } if len(limits) > 0 { - rt.Availability = router.QuotaAvailability{Store: s, Limits: limits, Window: 7 * 24 * time.Hour} + rt.Availability = router.QuotaAvailability{Store: s, Limits: limits} } if repo, root := os.Getenv("ORCHESTRA_REPO"), os.Getenv("ORCHESTRA_WORKTREE_ROOT"); repo != "" && root != "" { adapters := map[string]herdr.Adapter{} for _, h := range rr.Herdrs() { - if h.Address == "" { + address := herdrAddress(rr, h) + if address == "" { continue } - client := herdr.New(h.Address) + client := herdr.New(address) protocol := h.Protocol if protocol == "" { protocol = os.Getenv("ORCHESTRA_HERDR_PROTOCOL") @@ -77,7 +103,17 @@ func main() { log.Printf("herdr %s has unsupported harness %q", h.ID, h.Harness) } } - coordinator := &orchestrator.Coordinator{Store: s, StatePath: filepath.Join(dir, "runtime-sessions.json"), Worktrees: orchestrator.GitWorktrees{Root: root, Repo: repo}, Adapters: orchestrator.AdapterFactory{Herdrs: adapters}} + projectRepos := map[string]orchestrator.ProjectRepo{} + for _, p := range rr.Projects() { + if p.Repo != "" && p.WorktreeRoot != "" { + projectRepos[p.ID] = orchestrator.ProjectRepo{Repo: p.Repo, WorktreeRoot: p.WorktreeRoot} + } + } + worktrees := orchestrator.PerProjectGitWorktrees{ + Projects: projectRepos, + Default: orchestrator.GitWorktrees{Root: root, Repo: repo}, + } + coordinator = &orchestrator.Coordinator{Store: s, StatePath: filepath.Join(dir, "runtime-sessions.json"), Worktrees: worktrees, Adapters: orchestrator.AdapterFactory{Herdrs: adapters}} rt.OnLease = func(e domain.Event) error { return coordinator.Start(context.Background(), e) } hard := 0.75 if v, parseErr := strconv.ParseFloat(os.Getenv("ORCHESTRA_OCCUPANCY_HARD"), 64); parseErr == nil && v > 0 && v < 1 { @@ -96,7 +132,7 @@ func main() { for _, t := range s.Tasks() { if t.State == domain.StateLeased && t.Lease != nil && t.Lease.HarnessID == w.ID { p, _ := json.Marshal(map[string]any{"reason": "worker_offline", "harness_id": w.ID}) - e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p} + e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)} if err := s.Append(e); err == nil && rt != nil { _, _ = rt.HandleEvent(e) } @@ -126,7 +162,7 @@ func main() { return } b, _ := json.Marshal(p) - e := domain.Event{ID: id(), Type: "TaskCreated", TaskID: id(), Version: 1, Payload: b} + e := domain.Event{ID: id(), Type: "TaskCreated", TaskID: id(), Version: 1, Payload: b, Surface: string(surface(r))} if err := s.Append(e); err != nil { http.Error(w, err.Error(), 400) return @@ -147,6 +183,24 @@ func main() { } json.NewEncoder(w).Encode(s.Events(n)) }) + mux.HandleFunc("/v1/handoffs", func(w http.ResponseWriter, r *http.Request) { + out := make([]domain.Event, 0) + for _, e := range s.Events(0) { + if e.Type == "TaskReleased" { + out = append(out, e) + } + } + json.NewEncoder(w).Encode(out) + }) + mux.HandleFunc("/v1/quotas", func(w http.ResponseWriter, r *http.Request) { + out := make([]domain.Event, 0) + for _, e := range s.Events(0) { + if e.Type == "QuotaReported" { + out = append(out, e) + } + } + json.NewEncoder(w).Encode(out) + }) mux.HandleFunc("/v1/artifacts", func(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodPost { http.Error(w, "method not allowed", http.StatusMethodNotAllowed) @@ -223,7 +277,7 @@ func main() { return } b, _ := json.Marshal(map[string]any{"subject_ref": p.AdvisoryID}) - e := domain.Event{ID: id(), Type: "ApprovalGranted", TaskID: "system", Version: 0, Payload: b} + e := domain.Event{ID: id(), Type: "ApprovalGranted", TaskID: "system", Version: 0, Payload: b, Surface: string(surface(r))} if err := s.Append(e); err != nil { http.Error(w, err.Error(), 400) return @@ -256,7 +310,16 @@ func main() { adminServer := &admin.Server{Store: s, RouterReady: rt != nil, Providers: providerHealth, Probes: map[string]admin.ProbeFunc{ "router": func() (bool, string) { return rt != nil, "configured router" }, "gitea": func() (bool, string) { - return os.Getenv("ORCHESTRA_GITEA_URL") == "" || providerHealth["gitea"] != nil, "configured provider" + configured := os.Getenv("ORCHESTRA_GITEA_URL") != "" || os.Getenv("ORCHESTRA_GITEA_CONFIG") != "" + if !configured { + return true, "configured provider" + } + for name := range providerHealth { + if name == "gitea" || strings.HasPrefix(name, "gitea:") { + return true, "configured provider" + } + } + return false, "configured provider" }, "jsonl": func() (bool, string) { return os.Getenv("ORCHESTRA_JSONL") == "" || providerHealth["jsonl"] != nil, "configured provider" @@ -267,6 +330,26 @@ func main() { mux.HandleFunc("/v1/admin/diagnostics", adminServer.Diagnostics) mux.HandleFunc("/v1/tasks/", func(w http.ResponseWriter, r *http.Request) { parts := strings.Split(strings.Trim(r.URL.Path, "/"), "/") + if r.Method == http.MethodGet && coordinator != nil && len(parts) == 4 { + taskID, view := parts[2], parts[3] + if view == "health" { + if h, ok := coordinator.MonitorHealth().Sessions[taskID]; ok { + json.NewEncoder(w).Encode(h) + return + } + http.Error(w, "session health not found", 404) + return + } + if view == "capture" { + body, err := coordinator.Capture(r.Context(), taskID, r.URL.Query().Get("source")) + if err != nil { + http.Error(w, err.Error(), 404) + return + } + json.NewEncoder(w).Encode(map[string]string{"task_id": taskID, "source": r.URL.Query().Get("source"), "text": body}) + return + } + } if len(parts) < 4 || len(parts) > 5 || r.Method != "POST" { http.Error(w, "not found", http.StatusNotFound) return @@ -300,7 +383,7 @@ func main() { by = "surface" } b, _ := json.Marshal(map[string]any{"subject_ref": taskID, "by": by}) - e := domain.Event{ID: id(), Type: typ, TaskID: taskID, Version: t.Version + 1, Payload: b} + e := domain.Event{ID: id(), Type: typ, TaskID: taskID, Version: t.Version + 1, Payload: b, Surface: string(surface(r))} if err := s.Append(e); err != nil { http.Error(w, err.Error(), 409) return @@ -321,7 +404,7 @@ func main() { } b, _ := json.Marshal(map[string]any{"subject_ref": taskID, "options": p.Options}) t, _ := s.Task(taskID) - e := domain.Event{ID: id(), Type: "ApprovalRequested", TaskID: taskID, Version: t.Version + 1, Payload: b} + e := domain.Event{ID: id(), Type: "ApprovalRequested", TaskID: taskID, Version: t.Version + 1, Payload: b, Surface: string(surface(r))} if err := s.Append(e); err != nil { http.Error(w, err.Error(), 400) return @@ -377,7 +460,7 @@ func main() { return } ePayload, _ := json.Marshal(p) - e = domain.Event{ID: id(), Type: types[action], TaskID: taskID, Version: t.Version + 1, Payload: ePayload} + e = domain.Event{ID: id(), Type: types[action], TaskID: taskID, Version: t.Version + 1, Payload: ePayload, Surface: string(surface(r))} err = s.Append(e) default: http.Error(w, "unknown action", 404) @@ -416,6 +499,13 @@ func main() { } json.NewEncoder(w).Encode(out) }) + mux.HandleFunc("/v1/orchestrator/health", func(w http.ResponseWriter, r *http.Request) { + if coordinator == nil { + http.Error(w, "orchestrator unavailable", http.StatusServiceUnavailable) + return + } + json.NewEncoder(w).Encode(coordinator.MonitorHealth()) + }) mux.HandleFunc("/v1/federation/workers", func(w http.ResponseWriter, r *http.Request) { if r.Method == http.MethodGet { json.NewEncoder(w).Encode(workers.Snapshot()) @@ -521,6 +611,7 @@ func main() { TaskID string `json:"task_id"` TTLSeconds int `json:"ttl_seconds"` HandoffRef string `json:"handoff_ref"` + AnchorSHA string `json:"anchor_sha"` } if json.NewDecoder(r.Body).Decode(&b) != nil || b.TaskID == "" { http.Error(w, "invalid lease body", 400) @@ -551,8 +642,12 @@ func main() { http.Error(w, "handoff_ref required", 400) return } - p, _ := json.Marshal(map[string]any{"handoff_ref": b.HandoffRef, "harness_id": parts[3]}) - e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: b.TaskID, Version: t.Version + 1, Payload: p} + if len(b.AnchorSHA) != 40 { + http.Error(w, "anchor_sha required", 400) + return + } + p, _ := json.Marshal(map[string]any{"handoff_ref": b.HandoffRef, "harness_id": parts[3], "anchor_sha": b.AnchorSHA}) + e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: b.TaskID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)} if err := s.Append(e); err != nil { http.Error(w, err.Error(), 409) return @@ -562,25 +657,57 @@ func main() { } json.NewEncoder(w).Encode(e) }) - if base := os.Getenv("ORCHESTRA_GITEA_URL"); base != "" { - g := provider.Gitea{BaseURL: base, Token: os.Getenv("ORCHESTRA_GITEA_TOKEN"), WebhookSecret: os.Getenv("ORCHESTRA_GITEA_WEBHOOK_SECRET"), Owner: os.Getenv("ORCHESTRA_GITEA_OWNER"), Repo: os.Getenv("ORCHESTRA_GITEA_REPO")} - reflecting := provider.ReflectingSink{Sink: s, Tasks: s, Reflector: g} - mux.Handle("/v1/providers/gitea/webhook", g.WebhookHandler(reflecting)) - sup := &provider.Supervisor{Name: "gitea", Run: func(ctx context.Context) error { - pollCtx, cancel := context.WithTimeout(ctx, time.Minute) - defer cancel() - _, err := g.Poll(pollCtx, reflecting) - if err == nil { - select { - case <-ctx.Done(): - return ctx.Err() - case <-time.After(time.Minute): - } - } - return err + var giteaSources []provider.GiteaSourceConfig + if path := os.Getenv("ORCHESTRA_GITEA_CONFIG"); path != "" { + cfgs, err := provider.LoadGiteaConfigs(path) + if err != nil { + log.Fatalf("load gitea config: %v", err) + } + giteaSources = cfgs + } else if base := os.Getenv("ORCHESTRA_GITEA_URL"); base != "" { + // Legacy single-repo configuration: project defaults to the repo + // name, matching the historical (pre-multi-source) behavior. + giteaSources = []provider.GiteaSourceConfig{{ + Project: os.Getenv("ORCHESTRA_GITEA_REPO"), BaseURL: base, + Owner: os.Getenv("ORCHESTRA_GITEA_OWNER"), Repo: os.Getenv("ORCHESTRA_GITEA_REPO"), + Token: os.Getenv("ORCHESTRA_GITEA_TOKEN"), WebhookSecret: os.Getenv("ORCHESTRA_GITEA_WEBHOOK_SECRET"), }} - sup.Start(context.Background()) - providerHealth["gitea"] = sup + } + if len(giteaSources) > 0 { + reflectors := map[string]provider.Gitea{} + for _, c := range giteaSources { + g := provider.Gitea{BaseURL: c.BaseURL, Token: c.Token, WebhookSecret: c.WebhookSecret, Owner: c.Owner, Repo: c.Repo, Project: c.Project} + reflectors[g.SourceName()] = g + } + reflecting := provider.ReflectingSink{Sink: s, Tasks: s, Reflector: provider.MultiGitea{Sources: reflectors}} + for _, c := range giteaSources { + g := provider.Gitea{BaseURL: c.BaseURL, Token: c.Token, WebhookSecret: c.WebhookSecret, Owner: c.Owner, Repo: c.Repo, Project: c.Project} + name := "gitea:" + c.Project + webhookPath := "/v1/providers/gitea/webhook/" + c.Project + if len(giteaSources) == 1 && os.Getenv("ORCHESTRA_GITEA_CONFIG") == "" { + // Preserve the legacy unprefixed webhook path when running + // the single-source (env-var) configuration, so existing + // Gitea webhook configs don't need to be re-pointed. + webhookPath = "/v1/providers/gitea/webhook" + name = "gitea" + } + mux.Handle(webhookPath, g.WebhookHandler(reflecting)) + sup := &provider.Supervisor{Name: name, Run: func(ctx context.Context) error { + pollCtx, cancel := context.WithTimeout(ctx, time.Minute) + defer cancel() + _, err := g.Poll(pollCtx, reflecting) + if err == nil { + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(time.Minute): + } + } + return err + }} + sup.Start(context.Background()) + providerHealth[name] = sup + } } if path := os.Getenv("ORCHESTRA_JSONL"); path != "" { sup := &provider.Supervisor{Name: "jsonl", Run: func(ctx context.Context) error { @@ -620,6 +747,11 @@ func main() { } } }() + if rt != nil { + if _, err := rt.AssignPending(); err != nil { + log.Printf("startup task assignment: %v", err) + } + } port := os.Getenv("ORCHESTRA_PORT") if port == "" { port = "9145" diff --git a/deploy/config.example.json b/deploy/config.example.json new file mode 100644 index 0000000..6515589 --- /dev/null +++ b/deploy/config.example.json @@ -0,0 +1,66 @@ +{ + "projects": [ + { + "id": "correx", + "machine_affinity": ["mainframe"], + "repo": "/var/lib/orchestra/repos/correx.git", + "worktree_root": "/var/lib/orchestra/worktrees/correx" + }, + { + "id": "maven", + "machine_affinity": ["mainframe", "satellite"] + } + ], + "machines": [ + { + "id": "mainframe", + "address": "10.0.0.10:9145" + }, + { + "id": "satellite", + "address": "10.0.0.11:9145" + } + ], + "herdrs": [ + { + "id": "mainframe-claude-1", + "machine_id": "mainframe", + "harness": "claude", + "protocol": "1", + "capabilities": ["code", "review"], + "concurrency": 2, + "quota_limit_5h": 50, + "quota_limit_weekly": 500 + }, + { + "id": "satellite-claude-1", + "machine_id": "satellite", + "address": "10.0.0.11:9245", + "harness": "claude", + "protocol": "1", + "capabilities": ["code"], + "concurrency": 1, + "quota_limit_5h": 20, + "quota_limit_weekly": 200 + }, + { + "id": "mainframe-codex-1", + "machine_id": "mainframe", + "harness": "codex", + "protocol": "1", + "capabilities": ["code"], + "concurrency": 1, + "quota_limit_weekly": 300 + }, + { + "id": "satellite-opencode-1", + "machine_id": "satellite", + "address": "10.0.0.11:9345", + "harness": "opencode", + "protocol": "1", + "capabilities": ["code", "review"], + "concurrency": 1, + "quota_limit_weekly": 300 + } + ] +} diff --git a/deploy/config.example.jsonc b/deploy/config.example.jsonc new file mode 100644 index 0000000..f6eca44 --- /dev/null +++ b/deploy/config.example.jsonc @@ -0,0 +1,78 @@ +// Annotated reference for config.example.json (registry.Config, internal/registry/registry.go). +// This file is NOT valid JSON (it has comments) and is not loaded by orchestra — +// it exists purely to document fields. Copy config.example.json, not this file. +{ + // Static project topology. One entry per project the fleet routes tasks for. + "projects": [ + { + "id": "correx", // Project id; tasks/events are tagged with this. + "machine_affinity": ["mainframe"], // Machine ids (below) this project may run on. + // Required — a project with no affinity can't be routed. + "repo": "/var/lib/orchestra/repos/correx.git", // Optional per-project git repo path. + // Overrides the global ORCHESTRA_REPO default. + "worktree_root": "/var/lib/orchestra/worktrees/correx" // Optional per-project worktree dir. + // Overrides global ORCHESTRA_WORKTREE_ROOT. + }, + { + "id": "maven", + "machine_affinity": ["mainframe", "satellite"] // Multiple affinities: routable to either machine. + // repo/worktree_root omitted here: falls back to the deployment's global default. + } + ], + + // Physical/logical machines in the fleet. herdrs.machine_id below must reference one of these. + "machines": [ + { "id": "mainframe", "address": "10.0.0.10:9145" }, // address: host:port this machine's orchestra API listens on. + { "id": "satellite", "address": "10.0.0.11:9145" } + ], + + // Herdrs: individual harness worker slots that execute tasks. + "herdrs": [ + { + "id": "mainframe-claude-1", // Unique herdr id. + "machine_id": "mainframe", // Which machine (above) this herdr runs on. + // "address" omitted: falls back to the parent machine's address (used here since + // this herdr's harness listens on the machine's default port). + "harness": "claude", // Harness adapter to use: "claude" | "codex" | "opencode". + "protocol": "1", // Herdr wire protocol version. Falls back to + // ORCHESTRA_HERDR_PROTOCOL if omitted. + "capabilities": ["code", "review"], // Task capability tags this herdr can accept. + "concurrency": 2, // Max simultaneous sessions this herdr will run. + "quota_limit_5h": 50, // Rolling 5-hour usage quota (harness-specific units). + "quota_limit_weekly": 500 // Rolling weekly usage quota. + // "quota_limit" (deprecated): if set without quota_limit_5h, treated as weekly-only, + // to preserve old configs' historical meaning without inventing a 5h cap. + }, + { + "id": "satellite-claude-1", + "machine_id": "satellite", + "address": "10.0.0.11:9245", // Explicit override: this herdr listens on a non-default + // port on its machine (e.g. multiple herdrs per machine). + "harness": "claude", + "protocol": "1", + "capabilities": ["code"], + "concurrency": 1, + "quota_limit_5h": 20, + "quota_limit_weekly": 200 + }, + { + "id": "mainframe-codex-1", + "machine_id": "mainframe", + "harness": "codex", // OpenAI Codex CLI harness adapter. + "protocol": "1", + "capabilities": ["code"], + "concurrency": 1, + "quota_limit_weekly": 300 // codex has no separate 5h window tracked here; weekly only. + }, + { + "id": "satellite-opencode-1", + "machine_id": "satellite", + "address": "10.0.0.11:9345", + "harness": "opencode", // OpenCode CLI harness adapter. + "protocol": "1", + "capabilities": ["code", "review"], + "concurrency": 1, + "quota_limit_weekly": 300 + } + ] +} diff --git a/deploy/gitea.example.json b/deploy/gitea.example.json new file mode 100644 index 0000000..92ff444 --- /dev/null +++ b/deploy/gitea.example.json @@ -0,0 +1,18 @@ +[ + { + "project": "correx", + "base_url": "https://gitea.example.internal", + "owner": "kami", + "repo": "correx", + "token": "REPLACE_ME", + "webhook_secret": "REPLACE_ME" + }, + { + "project": "maven", + "base_url": "https://gitea.example.internal", + "owner": "kami", + "repo": "maven", + "token": "REPLACE_ME", + "webhook_secret": "REPLACE_ME" + } +] diff --git a/deploy/orchestra.env.example b/deploy/orchestra.env.example new file mode 100644 index 0000000..8d34488 --- /dev/null +++ b/deploy/orchestra.env.example @@ -0,0 +1,69 @@ +# Copy to /etc/orchestra/orchestra.env (chmod 600, owned by the orchestra +# user) and fill in the values you need. Referenced by orchestra.service via +# EnvironmentFile=. Every var below is read directly from os.Getenv in +# cmd/orchestra/main.go and the packages it wires up — grep ORCHESTRA_ in the +# repo if this list ever needs re-deriving. + +# --- Core --- +ORCHESTRA_DATA=/var/lib/orchestra/data +ORCHESTRA_PORT=9145 + +# Static project/machine/herdr topology (registry.Load). Required for +# routing across more than one machine; validated at startup. +ORCHESTRA_CONFIG=/etc/orchestra/config.json + +# --- Git worktrees (global default; per-project repo/worktree_root in +# ORCHESTRA_CONFIG overrides this per project — see registry.Project) --- +ORCHESTRA_REPO=/var/lib/orchestra/repo.git +ORCHESTRA_WORKTREE_ROOT=/var/lib/orchestra/worktrees + +# Protocol version fallback for herdrs that don't set "protocol" in +# ORCHESTRA_CONFIG. Prefer setting it per-herdr in the config; only use this +# if every herdr on the fleet truly matches. +#ORCHESTRA_HERDR_PROTOCOL=1 + +# Hard rotation occupancy threshold (0 < x < 1). Default 0.75 if unset/invalid. +ORCHESTRA_OCCUPANCY_HARD=0.75 + +# --- Providers --- +# Local JSONL task ingestion (baseline adapter). +#ORCHESTRA_JSONL=/var/lib/orchestra/tasks.jsonl + +# Gitea issue ingestion + terminal-state reflection. +# +# Multiple repos (one per project) — preferred if you have more than one +# Gitea-backed project. Points at a JSON array of +# {project,base_url,owner,repo,token,webhook_secret}; project is the +# registry project id ingested tasks are tagged with. Each source gets its +# own webhook path: /v1/providers/gitea/webhook/{project}. +#ORCHESTRA_GITEA_CONFIG=/etc/orchestra/gitea.json +# +# Single repo (legacy) — all four required together. Ignored if +# ORCHESTRA_GITEA_CONFIG is set. Webhook path is the unprefixed +# /v1/providers/gitea/webhook. Ingested tasks are tagged with project = +# ORCHESTRA_GITEA_REPO. +#ORCHESTRA_GITEA_URL=https://gitea.example.internal +#ORCHESTRA_GITEA_TOKEN= +#ORCHESTRA_GITEA_OWNER= +#ORCHESTRA_GITEA_REPO= +#ORCHESTRA_GITEA_WEBHOOK_SECRET= + +# --- Delivery (notify-only surfaces) --- +# Telegram: both required together. +#ORCHESTRA_TELEGRAM_BOT_TOKEN= +#ORCHESTRA_TELEGRAM_CHAT_ID= +# ntfy: topic required, token/url optional (self-hosted ntfy). +#ORCHESTRA_NTFY_TOPIC= +#ORCHESTRA_NTFY_TOKEN= +#ORCHESTRA_NTFY_URL=https://ntfy.sh + +# --- Bus authorization tokens (bearer auth per surface; a surface with no +# token set has no auth requirement — set these once you have real clients) --- +#ORCHESTRA_TUI_TOKEN= +#ORCHESTRA_WEB_TOKEN= +#ORCHESTRA_MCP_TOKEN= +#ORCHESTRA_MAVEN_TOKEN= +# Telegram/ntfy tokens above double as their surface auth tokens +# (ORCHESTRA_TELEGRAM_TOKEN is the inbound bearer token if you also expose an +# endpoint they poll, separate from the bot token used to send messages). +#ORCHESTRA_TELEGRAM_TOKEN= diff --git a/deploy/orchestra.service b/deploy/orchestra.service index 9c893f7..1e56db6 100644 --- a/deploy/orchestra.service +++ b/deploy/orchestra.service @@ -6,16 +6,20 @@ Wants=network-online.target [Service] Type=simple User=orchestra +Group=orchestra WorkingDirectory=/var/lib/orchestra -Environment=ORCHESTRA_DATA=/var/lib/orchestra/data -Environment=ORCHESTRA_PORT=9145 +EnvironmentFile=/etc/orchestra/orchestra.env ExecStart=/usr/local/bin/orchestra Restart=on-failure RestartSec=5s NoNewPrivileges=true ProtectSystem=strict -ReadWritePaths=/var/lib/orchestra +ProtectHome=true PrivateTmp=true +ReadWritePaths=/var/lib/orchestra +# ORCHESTRA_WORKTREE_ROOT / per-project worktree_root paths and +# ORCHESTRA_REPO must live under one of these, or under /var/lib/orchestra — +# add further ReadWritePaths= lines here if you keep repos elsewhere. [Install] WantedBy=multi-user.target diff --git a/deploy/redeploy.sh b/deploy/redeploy.sh new file mode 100755 index 0000000..c4eb98f --- /dev/null +++ b/deploy/redeploy.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)" +tmp_bin="$(mktemp)" +trap 'rm -f -- "$tmp_bin"' EXIT + +cd "$repo_dir" +echo "Building Orchestra..." +go build -o "$tmp_bin" ./cmd/orchestra + +echo "Installing /usr/local/bin/orchestra..." +sudo install -o root -g root -m 0755 "$tmp_bin" /usr/local/bin/orchestra + +echo "Restarting orchestra.service..." +sudo systemctl restart orchestra.service +sudo systemctl --no-pager --lines=8 status orchestra.service diff --git a/internal/admin/admin_test.go b/internal/admin/admin_test.go index 789c847..4467927 100644 --- a/internal/admin/admin_test.go +++ b/internal/admin/admin_test.go @@ -73,7 +73,7 @@ func TestSubscribeEmitsCursorAndEvent(t *testing.T) { t.Fatal(err) } p, _ := json.Marshal(map[string]any{"source": "test", "external_id": "1", "project": "p"}) - if err := s.Append(domain.Event{ID: "e", TaskID: "t", Type: "TaskCreated", Version: 1, Payload: p}); err != nil { + if err := s.Append(domain.Event{ID: "e", TaskID: "t", Type: "TaskCreated", Version: 1, Payload: p, Surface: string(authz.System)}); err != nil { t.Fatal(err) } h := (&Server{Store: s}).Subscribe diff --git a/internal/authz/authz.go b/internal/authz/authz.go index 59b4c20..340040b 100644 --- a/internal/authz/authz.go +++ b/internal/authz/authz.go @@ -17,6 +17,13 @@ const ( Web Surface = "web" MCP Surface = "mcp" Maven Surface = "maven" + // System identifies the plane itself — the router, coordinator, provider + // adapters, and lease-expiry reclaim. Per invariant 2 ("the plane emits + // events, not the agent"), these are the only non-surface emitters and are + // always full control. Every event must carry an explicit Surface; there + // is no unauthenticated default, so an emitter that forgets to declare one + // is rejected at the bus rather than silently treated as trusted. + System Surface = "system" ) type Capability int @@ -33,7 +40,7 @@ func CapabilityFor(s Surface) Capability { switch s { case Telegram, Ntfy: return NotifyOnly - case TUI, Web: + case TUI, Web, System: return FullControl case MCP, Maven: return GatedWrite diff --git a/internal/domain/domain.go b/internal/domain/domain.go index 78c22c8..9e19c3d 100644 --- a/internal/domain/domain.go +++ b/internal/domain/domain.go @@ -17,7 +17,11 @@ var ErrConflict = errors.New("task version conflict") var ErrNotFound = errors.New("task not found") var ErrInvalid = errors.New("invalid event") -const CurrentEventSchema = 1 +// CurrentEventSchema is 2: schema 2 requires every event to declare its +// authorizing Surface (see ValidateEvent), enforced at the store append +// boundary. Schema 1 events already on disk replay unchanged — tolerant +// reader, not upcast (spec open question #2). +const CurrentEventSchema = 2 type TaskState string @@ -63,6 +67,12 @@ type Event struct { Version int `json:"version"` At time.Time `json:"at"` Payload json.RawMessage `json:"payload"` + // Surface identifies the bus capability the emitter is authorized under + // (see internal/authz). It is required on every event so authorization is + // enforced once, at the store append boundary, regardless of whether the + // emitter reached the store over HTTP, from the router, from a harness + // adapter, or from a provider. + Surface string `json:"surface"` } func Hash(v []byte) string { h := sha256.Sum256(v); return hex.EncodeToString(h[:]) } @@ -82,6 +92,9 @@ func ValidateEvent(e Event) error { if e.SchemaVersion > CurrentEventSchema || e.Type == "" || e.TaskID == "" || len(e.Payload) == 0 || len(e.Payload) > 64*1024 { return ErrInvalid } + if e.SchemaVersion >= 2 && strings.TrimSpace(e.Surface) == "" { + return fmt.Errorf("%w: surface required", ErrInvalid) + } allowed := map[string]bool{"TaskCreated": true, "TaskLeased": true, "TaskReleased": true, "TaskCompleted": true, "TaskFailed": true, "TaskBlocked": true, "ApprovalRequested": true, "ApprovalGranted": true, "ApprovalDenied": true, "TaskAmended": true, "QuotaReported": true, "StandupAdvisory": true} if !allowed[e.Type] { return fmt.Errorf("%w: unknown type %q", ErrInvalid, e.Type) diff --git a/internal/domain/fuzz_test.go b/internal/domain/fuzz_test.go new file mode 100644 index 0000000..1a1b1ca --- /dev/null +++ b/internal/domain/fuzz_test.go @@ -0,0 +1,149 @@ +package domain + +import ( + "encoding/json" + "testing" +) + +// eventTypesUnderTest is the full lifecycle vocabulary the spec (§9 item 5) +// requires validators for; a validator that panics or accepts garbage for any +// of these on adversarial input is a defect regardless of whether real +// producers happen to send well-formed payloads. +var eventTypesUnderTest = []string{ + "TaskCreated", "TaskLeased", "TaskReleased", "TaskCompleted", "TaskFailed", + "TaskBlocked", "ApprovalRequested", "ApprovalGranted", "ApprovalDenied", + "TaskAmended", "QuotaReported", "StandupAdvisory", +} + +// FuzzValidatePayload feeds arbitrary JSON object shapes at every known event +// type's validator and requires it to either return a typed ErrInvalid or +// accept — never panic. The seed corpus below exercises adjacent-to-valid and +// wildly-malformed shapes (wrong types, huge strings, nested structures, +// nulls, NaN-adjacent floats via JSON) for each type. +func FuzzValidatePayload(f *testing.F) { + seeds := []string{ + `{}`, + `null`, + `{"source":"jsonl","external_id":"1","project":"p"}`, + `{"source":123,"external_id":null,"project":[]}`, + `{"harness_id":"h1","ttl":60,"expected_version":1}`, + `{"harness_id":"h1","until_ns":1e300,"expected_version":1.5}`, + `{"handoff_ref":"` + fakeHash() + `","anchor_sha":"` + fakeSHA() + `"}`, + `{"handoff_ref":123,"anchor_sha":true}`, + `{"report_ref":"` + fakeHash() + `","receipt":{"harness_id":"h1","consumed":1}}`, + `{"report_ref":"","receipt":{}}`, + `{"reason":"x"}`, + `{"reason":123}`, + `{"blocker":"x","handoff_ref":"` + fakeHash() + `"}`, + `{"blocker":""}`, + `{"amendment":"x"}`, + `{"subject_ref":"x","options":["a","b"]}`, + `{"subject_ref":123,"options":null}`, + `{"harness_id":"h1","consumed":90.5}`, + `{"harness_id":"h1","consumed":-1}`, + `{"harness_id":"h1","consumed":"a lot"}`, + `{"items":["standup line"]}`, + `{"items":null}`, + `{"a":{"b":{"c":{"d":[1,2,3,{"e":"f"}]}}}}`, + `{"x":` + hugeString() + `}`, + } + for _, s := range seeds { + f.Add(s) + } + + f.Fuzz(func(t *testing.T, raw string) { + var p map[string]any + if err := json.Unmarshal([]byte(raw), &p); err != nil { + return // not a JSON object; ValidateEvent itself rejects non-objects before reaching ValidatePayload + } + for _, typ := range eventTypesUnderTest { + func() { + defer func() { + if r := recover(); r != nil { + t.Fatalf("ValidatePayload(%q, %s) panicked: %v", typ, raw, r) + } + }() + err := ValidatePayload(typ, p) + if err != nil && err != ErrInvalid { + // must still be a typed validation error, wrapping ErrInvalid + if !isInvalid(err) { + t.Fatalf("ValidatePayload(%q, %s) returned non-typed error: %v", typ, raw, err) + } + } + }() + } + }) +} + +// FuzzValidateEvent exercises the full envelope path (schema version, surface +// requirement, type allow-list, payload size/parseability) with arbitrary +// type names, surfaces, and payload bytes, proving no combination panics. +func FuzzValidateEvent(f *testing.F) { + f.Add("TaskCreated", "system", 2, []byte(`{"source":"jsonl","external_id":"1","project":"p"}`)) + f.Add("", "", 0, []byte(``)) + f.Add("Bogus", "system", 2, []byte(`{}`)) + f.Add("TaskCreated", "", 2, []byte(`{"source":"jsonl","external_id":"1","project":"p"}`)) + f.Add("TaskCreated", "system", 1, []byte(`not json`)) + f.Add("QuotaReported", "system", 2, []byte(`null`)) + f.Add("TaskCompleted", "system", 99, []byte(`{"report_ref":"x","receipt":{}}`)) + + f.Fuzz(func(t *testing.T, typ, surface string, schema int, payload []byte) { + defer func() { + if r := recover(); r != nil { + t.Fatalf("ValidateEvent panicked: type=%q surface=%q schema=%d payload=%q: %v", typ, surface, schema, payload, r) + } + }() + e := Event{ + SchemaVersion: schema, + Type: typ, + TaskID: "t1", + Payload: payload, + Surface: surface, + } + _ = ValidateEvent(e) + }) +} + +func isInvalid(err error) bool { + for e := err; e != nil; { + if e == ErrInvalid { + return true + } + u, ok := e.(interface{ Unwrap() error }) + if !ok { + return false + } + e = u.Unwrap() + } + return false +} + +func fakeHash() string { + b := make([]byte, 32) + for i := range b { + b[i] = byte(i) + } + s := "" + for _, c := range b { + s += string("0123456789abcdef"[c>>4]) + string("0123456789abcdef"[c&0xf]) + } + return s +} + +func fakeSHA() string { + s := "" + for i := 0; i < 40; i++ { + s += "a" + } + return s +} + +func hugeString() string { + b, _ := json.Marshal(make([]byte, 0)) + _ = b + s := `"` + for i := 0; i < 5000; i++ { + s += "x" + } + return s + `"` +} diff --git a/internal/herdr/adapter.go b/internal/herdr/adapter.go index 94a1e67..970e786 100644 --- a/internal/herdr/adapter.go +++ b/internal/herdr/adapter.go @@ -2,7 +2,9 @@ package herdr import ( "context" + "encoding/json" "fmt" + "path/filepath" "strings" "time" ) @@ -15,6 +17,10 @@ type Adapter interface { Occupancy(Session) (float64, error) } +type WorktreeCreator interface { + CreateWorktree(context.Context, string, string, string) (string, error) +} + // TurnBoundary is optional so older herdr deployments remain usable. A true // result means the current harness turn has ended and handoff is safe. type TurnBoundary interface { @@ -26,6 +32,18 @@ type RotationSignal interface { type PaneExit interface { PaneExited(context.Context, Session) (bool, error) } + +// AgentStatus is a live, non-lifecycle status reported by herdr. Consumers +// must not infer task completion or release from it. +type AgentStatus interface { + AgentStatus(context.Context, Session) (string, error) +} +type AgentBlocker interface { + AgentBlocker(context.Context, Session) (string, error) +} +type PaneCapture interface { + PaneCapture(context.Context, Session, string) (string, error) +} type CLIAdapter struct { Client *Client Harness string @@ -33,15 +51,29 @@ type CLIAdapter struct { Usage func(string) (Usage, error) } +func (a CLIAdapter) CreateWorktree(ctx context.Context, repo, root, taskID string) (string, error) { + path, err := a.Client.Worktree(ctx, repo, filepath.Join(root, taskID), "orchestra/"+taskID) + if err != nil { + return "", err + } + if path == "" { + return "", fmt.Errorf("adapter: herdr returned empty worktree path") + } + return path, nil +} + func (a CLIAdapter) Lease(ctx context.Context, task, worktree string) (Session, error) { if a.Client == nil { return Session{}, fmt.Errorf("adapter: client required") } - var s Session - e := a.Client.Call(ctx, "pane.create", map[string]string{"harness": a.Harness, "task_id": task, "worktree": worktree}, &s) - s.Harness = a.Harness - s.Worktree = worktree - return s, e + s, err := a.Client.StartAgent(ctx, worktree, worktree, "orchestra/"+task, a.Harness, task) + if err != nil { + return Session{}, err + } + if err := a.Client.Prompt(ctx, s.PaneID, fmt.Sprintf("Begin Orchestra task %s. Inspect the repository, understand the task context, and proceed with the requested work.", task), 0); err != nil { + return Session{}, err + } + return s, nil } func (a CLIAdapter) Bootstrap(ctx context.Context, s Session, ref string) error { return a.Client.Prompt(ctx, s.PaneID, fmt.Sprintf("Read handoff %s, validate the anchor and TASK.md, then continue.", ref), time.Minute) @@ -57,23 +89,96 @@ func (a CLIAdapter) Kill(ctx context.Context, s Session) error { return a.Client.Call(ctx, "pane.kill", s, nil) } func (a CLIAdapter) AtTurnBoundary(ctx context.Context, s Session) (bool, error) { - var r struct { - Status string `json:"status"` - } - if err := a.Client.Call(ctx, "pane.status", s, &r); err != nil { + status, err := a.AgentStatus(ctx, s) + if err != nil { return false, err } - return !IsBusy(r.Status), nil + return !IsBusy(status), nil } func (a CLIAdapter) PaneExited(ctx context.Context, s Session) (bool, error) { - var r struct { - Status string `json:"status"` - } - if err := a.Client.Call(ctx, "pane.status", s, &r); err != nil { + status, err := a.AgentStatus(ctx, s) + if err != nil { return false, err } - return strings.EqualFold(r.Status, "exited") || strings.EqualFold(r.Status, "dead"), nil + return strings.EqualFold(status, "exited") || strings.EqualFold(status, "dead"), nil } +func (a CLIAdapter) AgentStatus(ctx context.Context, s Session) (string, error) { + // Current herdr protocol exposes agent state through agent.get; older + // Orchestra code used pane.status, which is not a valid protocol method. + var r map[string]any + if err := a.Client.Call(ctx, "agent.get", map[string]any{"target": s.PaneID}, &r); err != nil { + return "", err + } + return statusFromAgentResult(r), nil +} + +func (a CLIAdapter) AgentBlocker(ctx context.Context, s Session) (string, error) { + var r struct { + Read struct { + Text string `json:"text"` + } `json:"read"` + } + if err := a.Client.Call(ctx, "pane.read", map[string]any{"pane_id": s.PaneID, "source": "recent"}, &r); err != nil { + return "", err + } + text := strings.TrimSpace(r.Read.Text) + lines := strings.Split(text, "\n") + for i, raw := range lines { + line := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(raw), "┃")) + if !strings.EqualFold(line, "Permission required") && !strings.EqualFold(line, "Approval required") && !strings.HasPrefix(strings.ToLower(line), "waiting for") { + continue + } + for _, next := range lines[i+1:] { + command := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(next), "┃")) + if strings.HasPrefix(command, "$ ") { + return strings.ToLower(line) + ": shell command `" + strings.TrimSpace(strings.TrimPrefix(command, "$ ")) + "`", nil + } + } + return strings.ToLower(line), nil + } + return "", nil +} + +func (a CLIAdapter) PaneCapture(ctx context.Context, s Session, source string) (string, error) { + if source == "" { + source = "recent" + } + var r struct { + Read struct { + Text string `json:"text"` + } `json:"read"` + } + if err := a.Client.Call(ctx, "pane.read", map[string]any{"pane_id": s.PaneID, "source": source}, &r); err != nil { + return "", err + } + return r.Read.Text, nil +} + +func statusFromAgentResult(v any) string { + if m, ok := v.(map[string]any); ok { + for _, key := range []string{"status", "agent_status", "state"} { + if s, ok := m[key].(string); ok && s != "" { + return s + } + } + for _, child := range m { + if s := statusFromAgentResult(child); s != "" { + return s + } + } + } + if a, ok := v.([]any); ok { + for _, child := range a { + if s := statusFromAgentResult(child); s != "" { + return s + } + } + } + return "" +} + +var _ = json.RawMessage{} + func (a CLIAdapter) RotationSignal(ctx context.Context, s Session) (string, error) { var r struct { Reason string `json:"reason"` diff --git a/internal/herdr/herdr.go b/internal/herdr/herdr.go index aad565d..77c723c 100644 --- a/internal/herdr/herdr.go +++ b/internal/herdr/herdr.go @@ -13,6 +13,7 @@ import ( "os" "os/exec" "path/filepath" + "strings" "sync" "time" ) @@ -22,13 +23,15 @@ var ErrProtocol = errors.New("herdr protocol error") type Request struct { ID string `json:"id"` Method string `json:"method"` - Params any `json:"params,omitempty"` + // Herdr's JSON-RPC decoder requires params to be present, including for + // parameterless calls such as ping. Encode nil as an explicit JSON null. + Params any `json:"params"` } type Response struct { ID string `json:"id"` Result json.RawMessage `json:"result"` Error *struct { - Code int `json:"code"` + Code string `json:"code"` Message string `json:"message"` } `json:"error,omitempty"` } @@ -38,14 +41,39 @@ type Client struct { dial func() (net.Conn, error) mu sync.Mutex next uint64 + panes map[string]string } -func New(path string) *Client { return &Client{Path: path, Timeout: 10 * time.Second} } +type WorktreeInfo struct { + Path string `json:"path"` +} + +type worktreeResponse struct { + Path string `json:"path"` + Worktree WorktreeInfo `json:"worktree"` + RootPane struct { + PaneID string `json:"pane_id"` + Agent string `json:"agent"` + } `json:"root_pane"` + Workspace struct { + RootPane struct { + PaneID string `json:"pane_id"` + } `json:"root_pane"` + } `json:"workspace"` +} + +func New(path string) *Client { + return &Client{Path: path, Timeout: 10 * time.Second, panes: map[string]string{}} +} func (c *Client) conn() (net.Conn, error) { if c.dial != nil { return c.dial() } - return net.DialTimeout("unix", c.Path, c.Timeout) + network := "unix" + if strings.Contains(c.Path, "://") || (strings.Contains(c.Path, ":") && !strings.HasPrefix(c.Path, "/")) { + network = "tcp" + } + return net.DialTimeout(network, c.Path, c.Timeout) } func (c *Client) Call(ctx context.Context, method string, params any, out any) error { c.mu.Lock() @@ -62,6 +90,9 @@ func (c *Client) Call(ctx context.Context, method string, params any, out any) e } else if c.Timeout > 0 { _ = cn.SetDeadline(time.Now().Add(c.Timeout)) } + if params == nil { + params = map[string]any{} + } if err = json.NewEncoder(cn).Encode(Request{ID: id, Method: method, Params: params}); err != nil { return err } @@ -79,8 +110,8 @@ func (c *Client) Call(ctx context.Context, method string, params any, out any) e } type PingResult struct { - Protocol string `json:"protocol"` - Version string `json:"version"` + Protocol json.RawMessage `json:"protocol"` + Version json.RawMessage `json:"version"` } func (c *Client) Ping(ctx context.Context) (PingResult, error) { @@ -93,8 +124,14 @@ func (c *Client) CheckProtocol(ctx context.Context, want string) error { if e != nil { return e } - if want != "" && p.Protocol != want { - return fmt.Errorf("%w: want %s, got %s", ErrProtocol, want, p.Protocol) + if want != "" { + var text string + if err := json.Unmarshal(p.Protocol, &text); err != nil { + text = string(p.Protocol) + } + if text != want { + return fmt.Errorf("%w: want %s, got %s", ErrProtocol, want, text) + } } return nil } @@ -103,18 +140,80 @@ type Session struct { PaneID string `json:"pane_id"` Worktree string `json:"worktree"` Harness string `json:"harness"` + HerdrID string `json:"herdr_id,omitempty"` } func (c *Client) Prompt(ctx context.Context, pane, text string, wait time.Duration) error { - p := map[string]any{"pane_id": pane, "prompt": text, "wait": map[string]any{"until": "turn_end", "timeout_ms": wait.Milliseconds()}} + p := map[string]any{"target": pane, "text": text} + if wait > 0 { + p["wait"] = map[string]any{"until": []string{"idle"}, "timeout_ms": wait.Milliseconds()} + } return c.Call(ctx, "agent.prompt", p, nil) } -func (c *Client) Worktree(ctx context.Context, path, branch string) (string, error) { - var r struct { - Path string `json:"path"` +func (c *Client) Worktree(ctx context.Context, cwd, path, branch string) (string, error) { + var r worktreeResponse + // Protocol 17 requires exactly one of path or branch. Use the explicit + // path so the worker owns the checkout location; herdr manages the branch + // associated with that worktree. + p := map[string]any{"cwd": cwd, "path": path} + e := c.Call(ctx, "worktree.create", p, &r) + if e != nil && strings.Contains(strings.ToLower(e.Error()), "already exists") { + e = c.Call(ctx, "worktree.open", map[string]any{"cwd": cwd, "path": path}, &r) } - e := c.Call(ctx, "worktree.create", map[string]string{"path": path, "branch": branch}, &r) - return r.Path, e + if e != nil { + return "", e + } + if r.RootPane.PaneID != "" { + c.mu.Lock() + c.panes[path] = r.RootPane.PaneID + c.mu.Unlock() + } + if r.Path != "" { + return r.Path, nil + } + return r.Worktree.Path, nil +} + +func (c *Client) StartAgent(ctx context.Context, cwd, path, branch, harness, taskID string) (Session, error) { + c.mu.Lock() + paneID := c.panes[path] + c.mu.Unlock() + if paneID == "" { + return Session{}, fmt.Errorf("herdr: no pane recorded for worktree %s", path) + } + var s Session + if err := c.Call(ctx, "agent.start", map[string]any{ + "pane_id": paneID, + "kind": harness, + "name": harness, + "args": []string{}, + }, &s); err != nil { + if !strings.Contains(strings.ToLower(err.Error()), "already") { + return Session{}, err + } + } + s.PaneID = paneID + s.Worktree = path + s.Harness = harness + return s, nil +} + +// HeadSHA returns the current commit of a worktree. The rotation path uses +// this to populate TaskReleased.anchor_sha without trusting the adapter's +// opaque handoff-ref return value. +func HeadSHA(root string) (string, error) { + out, err := exec.Command("git", "-C", root, "rev-parse", "HEAD").Output() + if err != nil { + return "", err + } + sha := string(out) + if len(sha) > 0 && sha[len(sha)-1] == '\n' { + sha = sha[:len(sha)-1] + } + if len(sha) != 40 { + return "", fmt.Errorf("herdr: unexpected HEAD output %q", sha) + } + return sha, nil } // AnchorValid checks the split-then-close safety condition without trusting a diff --git a/internal/integration/end_to_end_test.go b/internal/integration/end_to_end_test.go index b9b8a46..ee1ff42 100644 --- a/internal/integration/end_to_end_test.go +++ b/internal/integration/end_to_end_test.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json" "errors" + "orchestra/internal/authz" "orchestra/internal/domain" "orchestra/internal/herdr" "orchestra/internal/orchestrator" @@ -128,7 +129,7 @@ func TestEndToEndIngestRouteLeaseRotateAndComplete(t *testing.T) { } got, _ := s.Task(task.ID) if got.State == domain.StateLeased && h.releases == 1 { - if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: got.Version + 1, Payload: mustJSON(map[string]string{ + if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: got.Version + 1, Surface: string(authz.System), Payload: mustJSON(map[string]string{ "handoff_ref": h.ref, "anchor_sha": "0123456789012345678901234567890123456789", })}); err != nil { @@ -153,7 +154,7 @@ func TestEndToEndIngestRouteLeaseRotateAndComplete(t *testing.T) { if err != nil { t.Fatal(err) } - if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskCompleted", TaskID: task.ID, Version: got.Version + 1, Payload: mustJSON(map[string]any{ + if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskCompleted", TaskID: task.ID, Version: got.Version + 1, Surface: string(authz.System), Payload: mustJSON(map[string]any{ "report_ref": ref, "receipt": map[string]any{"harness_id": "h1", "consumed": 1}, })}); err != nil { @@ -181,7 +182,7 @@ func TestRestartReplayAndReconcileKillsOrphan(t *testing.T) { } // A fresh coordinator sees the durable session, then drops it once the lease is gone. ref, _ := s.PutArtifact([]byte("handoff")) - if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: 3, Payload: mustJSON(map[string]string{ + if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: 3, Surface: string(authz.System), Payload: mustJSON(map[string]string{ "handoff_ref": ref, "anchor_sha": "0123456789012345678901234567890123456789", })}); err != nil { @@ -207,10 +208,10 @@ func TestRestartReplayAndReconcileKillsOrphan(t *testing.T) { func TestProviderRetryReflectionQuotaAndVersionConflict(t *testing.T) { s, r, _ := setup(t) task := ingest(t, s, "retry") - if err := s.Append(domain.Event{Type: "QuotaReported", TaskID: "quota", Version: 1, Payload: mustJSON(map[string]any{"harness_id": "h1", "consumed": 90.0})}); err != nil { + if err := s.Append(domain.Event{Type: "QuotaReported", TaskID: "quota", Version: 1, Surface: string(authz.System), Payload: mustJSON(map[string]any{"harness_id": "h1", "consumed": 90.0})}); err != nil { t.Fatal(err) } - rt := router.Router{Store: s, Registry: r, Reachability: alwaysReachable{}, Availability: router.QuotaAvailability{Store: s, Limits: map[string]float64{"h1": 100}, Now: time.Now}} + rt := router.Router{Store: s, Registry: r, Reachability: alwaysReachable{}, Availability: router.QuotaAvailability{Store: s, Limits: map[string]router.QuotaWindowLimits{"h1": {Weekly: 100}}, Now: time.Now}} if got, _ := rt.AssignPending(); len(got) != 0 { t.Fatalf("quota assigned=%d", len(got)) } @@ -225,7 +226,7 @@ func TestProviderRetryReflectionQuotaAndVersionConflict(t *testing.T) { t.Fatal(err) } reflector := &fakeReflector{} - if err := (provider.ReflectingSink{Sink: s, Tasks: s, Reflector: reflector}).Append(domain.Event{ID: domain.NewID(), Type: "TaskCompleted", TaskID: task.ID, Version: 3, Payload: mustJSON(map[string]any{ + if err := (provider.ReflectingSink{Sink: s, Tasks: s, Reflector: reflector}).Append(domain.Event{ID: domain.NewID(), Type: "TaskCompleted", TaskID: task.ID, Version: 3, Surface: string(authz.System), Payload: mustJSON(map[string]any{ "report_ref": ref, "receipt": map[string]any{"harness_id": "h1", "consumed": 1}, })}); err != nil { diff --git a/internal/integration/federation_lease_test.go b/internal/integration/federation_lease_test.go new file mode 100644 index 0000000..c2f24e5 --- /dev/null +++ b/internal/integration/federation_lease_test.go @@ -0,0 +1,213 @@ +package integration + +import ( + "context" + "encoding/json" + "orchestra/internal/authz" + "orchestra/internal/domain" + "orchestra/internal/federation" + "orchestra/internal/herdr" + "orchestra/internal/operations" + "orchestra/internal/orchestrator" + "orchestra/internal/registry" + "orchestra/internal/router" + "orchestra/internal/store" + "os/exec" + "testing" + "time" +) + +func gitInit(t *testing.T, dir, marker string) string { + t.Helper() + run := func(args ...string) { + cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("git %v: %v: %s", args, err, out) + } + } + run("init") + run("config", "user.email", "t@t") + run("config", "user.name", "t") + run("commit", "--allow-empty", "-m", "init: "+marker) + head, err := herdr.HeadSHA(dir) + if err != nil { + t.Fatal(err) + } + return head +} + +type fixedWorktree struct{ path string } + +func (w fixedWorktree) Create(context.Context, domain.Task) (string, error) { return w.path, nil } + +// machineAdapter is like harness but actually threads the worktree it was +// leased against into the Session, the way a real herdr adapter must — the +// shared harness fixture ignores it, which is fine for single-checkout +// tests but would silently defeat this one's anchor_sha assertions. +type machineAdapter struct{ *harness } + +func (a machineAdapter) Lease(_ context.Context, _ string, worktree string) (herdr.Session, error) { + return herdr.Session{Harness: "h1", PaneID: "pane-1", Worktree: worktree}, nil +} + +type singleAdapter struct{ a herdr.Adapter } + +func (s singleAdapter) Adapter(string) (herdr.Adapter, error) { return s.a, nil } + +// TestCrossMachineLeaseAnchorAndQuotaArePerHost exercises spec §2.1/§9 open +// question 8 end to end using two independent local git checkouts standing +// in for homesrv and workpc: each machine's Coordinator only ever validates +// and stamps anchor_sha from its OWN local checkout (never the other +// machine's), and quota consumption is accounted strictly per harness/host +// so one machine exhausting its window cannot block the other from being +// leased work. Federation registration/heartbeat plumbing (already covered +// by internal/federation's own tests) is exercised alongside this to prove +// the pieces fit together, not just in isolation. +func TestCrossMachineLeaseAnchorAndQuotaArePerHost(t *testing.T) { + homesrvRepo, workpcRepo := t.TempDir(), t.TempDir() + homesrvHead := gitInit(t, homesrvRepo, "homesrv") + workpcHead := gitInit(t, workpcRepo, "workpc") + if homesrvHead == workpcHead { + t.Fatal("test setup: expected distinct checkouts") + } + + s, err := store.Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + + // The federation registry is homesrv's view of workpc as an intermittent + // worker (spec §2.1): it must be registered and reachable before the + // router would ever consider leasing to it. + workers := &federation.Registry{} + if err := workers.Register(federation.Worker{ID: "workpc", Address: "workpc.mesh", Token: "secret"}); err != nil { + t.Fatal(err) + } + if err := workers.Heartbeat("workpc"); err != nil { + t.Fatal(err) + } + online := false + for _, w := range workers.Snapshot() { + if w.ID == "workpc" && w.Online { + online = true + } + } + if !online { + t.Fatal("workpc worker should be online after heartbeat") + } + + // Two tasks, one leased to each machine's harness. + mk := func(id string) domain.Task { + b, _ := json.Marshal(map[string]any{"source": "jsonl", "external_id": id, "project": "p"}) + if err := s.Append(domain.Event{ID: id, Type: "TaskCreated", TaskID: id, Version: 1, Surface: string(authz.System), Payload: b}); err != nil { + t.Fatal(err) + } + task, _ := s.Task(id) + return task + } + homesrvTask := mk("home-task") + workpcTask := mk("workpc-task") + + homesrvAdapter := &harness{occupancy: .95, boundary: true, ref: mustArtifact(t, s, "home-handoff")} + workpcAdapter := &harness{occupancy: .95, boundary: true, ref: mustArtifact(t, s, "workpc-handoff")} + + homesrvCoord := &orchestrator.Coordinator{Store: s, Worktrees: fixedWorktree{homesrvRepo}, Adapters: singleAdapter{machineAdapter{homesrvAdapter}}, StatePath: t.TempDir() + "/home-sessions.json"} + workpcCoord := &orchestrator.Coordinator{Store: s, Worktrees: fixedWorktree{workpcRepo}, Adapters: singleAdapter{machineAdapter{workpcAdapter}}, StatePath: t.TempDir() + "/workpc-sessions.json"} + + homesrvLease, err := s.Lease(homesrvTask.ID, "homesrv-h1", time.Minute) + if err != nil { + t.Fatal(err) + } + if err := homesrvCoord.Start(context.Background(), homesrvLease); err != nil { + t.Fatal(err) + } + workpcLease, err := s.Lease(workpcTask.ID, "workpc-h1", time.Minute) + if err != nil { + t.Fatal(err) + } + if err := workpcCoord.Start(context.Background(), workpcLease); err != nil { + t.Fatal(err) + } + + ctxHome, cancelHome := context.WithCancel(context.Background()) + defer cancelHome() + ctxWork, cancelWork := context.WithCancel(context.Background()) + defer cancelWork() + go homesrvCoord.Monitor(ctxHome, .8, time.Millisecond) + go workpcCoord.Monitor(ctxWork, .8, time.Millisecond) + + waitQueued := func(id string) domain.Task { + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if t, ok := s.Task(id); ok && t.State == domain.StateQueued { + return t + } + time.Sleep(time.Millisecond) + } + t.Fatalf("task %s never rotated back to queued", id) + return domain.Task{} + } + waitQueued(homesrvTask.ID) + waitQueued(workpcTask.ID) + + // Each machine's coordinator must have stamped anchor_sha from its OWN + // checkout — never the other machine's HEAD, and never each other's. + anchorFor := func(taskID string) string { + for _, e := range s.Events(0) { + if e.TaskID != taskID || e.Type != "TaskReleased" { + continue + } + var p map[string]any + _ = json.Unmarshal(e.Payload, &p) + return p["anchor_sha"].(string) + } + t.Fatalf("no TaskReleased found for %s", taskID) + return "" + } + if got := anchorFor(homesrvTask.ID); got != homesrvHead { + t.Fatalf("homesrv anchor_sha=%s want=%s (must validate against its own checkout)", got, homesrvHead) + } + if got := anchorFor(workpcTask.ID); got != workpcHead { + t.Fatalf("workpc anchor_sha=%s want=%s (must validate against its own checkout, not homesrv's)", got, workpcHead) + } + + // Quota is accounted per host/harness: exhausting homesrv-h1 must not + // affect workpc-h1's availability, and vice versa (spec §2.1, §7.2). + quotaReport := func(harness string, consumed float64) { + p, _ := json.Marshal(map[string]any{"harness_id": harness, "consumed": consumed}) + if err := s.Append(domain.Event{ID: domain.NewID(), Type: "QuotaReported", TaskID: "quota", Version: 1, Surface: string(authz.System), Payload: p}); err != nil { + t.Fatal(err) + } + } + quotaReport("homesrv-h1", 95) + limits := map[string]router.QuotaWindowLimits{ + "homesrv-h1": {Weekly: 100}, + "workpc-h1": {Weekly: 100}, + } + avail := router.QuotaAvailability{Store: s, Limits: limits} + if avail.Available(registry.Herdr{ID: "homesrv-h1"}) { + t.Fatal("homesrv-h1 should be quota-exhausted at 95/100") + } + if !avail.Available(registry.Herdr{ID: "workpc-h1"}) { + t.Fatal("workpc-h1 exhaustion leaked from homesrv-h1's per-host accounting") + } + + // The windowed quota aggregation used by the brief (spec §7.4) must also + // keep the two hosts separate. + sums := operations.AggregateQuota(s.Events(0), time.Now().Add(-time.Hour), time.Now().Add(time.Hour)) + if sums["homesrv-h1"] != 95 { + t.Fatalf("homesrv-h1 aggregate=%v want 95", sums["homesrv-h1"]) + } + if sums["workpc-h1"] != 0 { + t.Fatalf("workpc-h1 aggregate=%v want 0 (must not inherit homesrv-h1's receipts)", sums["workpc-h1"]) + } +} + +func mustArtifact(t *testing.T, s *store.Store, content string) string { + t.Helper() + ref, err := s.PutArtifact([]byte(content)) + if err != nil { + t.Fatal(err) + } + return ref +} diff --git a/internal/operations/operations.go b/internal/operations/operations.go index e862f52..a652c8a 100644 --- a/internal/operations/operations.go +++ b/internal/operations/operations.go @@ -4,6 +4,7 @@ package operations import ( "encoding/json" "fmt" + "orchestra/internal/authz" "orchestra/internal/domain" "orchestra/internal/store" "os/exec" @@ -99,7 +100,7 @@ func GenerateStandupAdvisory(s *store.Store, at time.Time) (domain.Event, error) if err != nil { return domain.Event{}, err } - e := domain.Event{ID: domain.NewID(), TaskID: "system", Type: "StandupAdvisory", Payload: p, At: at} + e := domain.Event{ID: domain.NewID(), TaskID: "system", Type: "StandupAdvisory", Payload: p, At: at, Surface: string(authz.System)} return e, s.Append(e) } @@ -145,7 +146,7 @@ func ApplyAdvisory(s *store.Store, advisoryID string) ([]domain.Event, error) { continue } b, _ := json.Marshal(map[string]any{"title": item.Title, "advisory_ref": advisoryID}) - e := domain.Event{ID: domain.NewID(), TaskID: item.TaskID, Type: "TaskAmended", Version: t.Version + 1, Payload: b} + e := domain.Event{ID: domain.NewID(), TaskID: item.TaskID, Type: "TaskAmended", Version: t.Version + 1, Payload: b, Surface: string(authz.System)} if err := s.Append(e); err != nil { return out, err } diff --git a/internal/operations/operations_test.go b/internal/operations/operations_test.go index 538a880..1e48cdd 100644 --- a/internal/operations/operations_test.go +++ b/internal/operations/operations_test.go @@ -2,6 +2,7 @@ package operations import ( "encoding/json" + "orchestra/internal/authz" "orchestra/internal/domain" "orchestra/internal/store" "testing" @@ -22,7 +23,7 @@ func TestAggregateQuotaSumsRotationsAndWindows(t *testing.T) { now := time.Now().UTC() enc := func(at time.Time, n float64) domain.Event { p, _ := json.Marshal(map[string]any{"harness_id": "codex", "consumed": n}) - return domain.Event{Type: "QuotaReported", At: at, Payload: p} + return domain.Event{Type: "QuotaReported", At: at, Payload: p, Surface: string(authz.System)} } es := []domain.Event{enc(now.Add(-2*time.Hour), 4), enc(now.Add(-time.Hour), 6), enc(now.Add(-48*time.Hour), 100)} got := AggregateQuota(es, now.Add(-3*time.Hour), now) @@ -37,11 +38,11 @@ func TestApplyAdvisoryRequiresApproval(t *testing.T) { t.Fatal(err) } p, _ := json.Marshal(map[string]any{"source": "test", "external_id": "1", "project": "p", "title": "old"}) - if err := s.Append(domain.Event{ID: "task", TaskID: "task", Type: "TaskCreated", Version: 1, Payload: p}); err != nil { + if err := s.Append(domain.Event{ID: "task", TaskID: "task", Type: "TaskCreated", Version: 1, Payload: p, Surface: string(authz.System)}); err != nil { t.Fatal(err) } ap, _ := json.Marshal(map[string]any{"items": []StandupItem{{TaskID: "task", Title: "new"}}}) - adv := domain.Event{ID: "adv", TaskID: "system", Type: "StandupAdvisory", Payload: ap} + adv := domain.Event{ID: "adv", TaskID: "system", Type: "StandupAdvisory", Payload: ap, Surface: string(authz.System)} if err := s.Append(adv); err != nil { t.Fatal(err) } @@ -49,7 +50,7 @@ func TestApplyAdvisoryRequiresApproval(t *testing.T) { t.Fatal("unapproved advisory applied") } grant, _ := json.Marshal(map[string]any{"subject_ref": "adv"}) - if err := s.Append(domain.Event{ID: "grant", TaskID: "system", Type: "ApprovalGranted", Payload: grant}); err != nil { + if err := s.Append(domain.Event{ID: "grant", TaskID: "system", Type: "ApprovalGranted", Payload: grant, Surface: string(authz.System)}); err != nil { t.Fatal(err) } if _, err := ApplyAdvisory(s, "adv"); err != nil { diff --git a/internal/orchestrator/orchestrator.go b/internal/orchestrator/orchestrator.go index 16db447..12b95b2 100644 --- a/internal/orchestrator/orchestrator.go +++ b/internal/orchestrator/orchestrator.go @@ -7,6 +7,7 @@ import ( "context" "encoding/json" "fmt" + "orchestra/internal/authz" "orchestra/internal/continuity" "orchestra/internal/domain" "orchestra/internal/herdr" @@ -14,6 +15,7 @@ import ( "os" "os/exec" "path/filepath" + "strings" "sync" "time" ) @@ -21,6 +23,12 @@ import ( type Worktrees interface { Create(context.Context, domain.Task) (string, error) } +type WorktreeSpec interface { + Spec(domain.Task) (string, string, bool) +} +type WorktreeCleaner interface { + Remove(context.Context, domain.Task, string) error +} type Adapters interface { Adapter(string) (herdr.Adapter, error) } @@ -34,6 +42,10 @@ type GitWorktrees struct { TaskFileSHA string } +func (w GitWorktrees) Spec(domain.Task) (string, string, bool) { + return w.Repo, w.Root, w.Repo != "" && w.Root != "" +} + func (w GitWorktrees) Create(ctx context.Context, t domain.Task) (string, error) { if w.Root == "" || w.Repo == "" { return "", fmt.Errorf("worktree: root and repo required") @@ -63,6 +75,62 @@ func (w GitWorktrees) Create(ctx context.Context, t domain.Task) (string, error) return p, nil } +func (w GitWorktrees) Remove(ctx context.Context, _ domain.Task, path string) error { + if path == "" { + return fmt.Errorf("worktree: path required") + } + cmd := exec.CommandContext(ctx, "git", "-C", w.Repo, "worktree", "remove", "--force", path) + if out, err := cmd.CombinedOutput(); err != nil { + return fmt.Errorf("%s: %w", string(out), err) + } + return nil +} + +// ProjectRepo is the minimal shape PerProjectGitWorktrees needs from a +// project's registry entry — kept local (not importing internal/registry) +// so orchestrator does not depend on registry's config-loading concerns. +type ProjectRepo struct { + Repo string + WorktreeRoot string +} + +// PerProjectGitWorktrees resolves a task's repo/root by its project (spec +// §2.2: each project is first-class and may have its own checkout), falling +// back to Default for any project not present in Projects — this keeps +// single-repo deployments working unchanged. +type PerProjectGitWorktrees struct { + Projects map[string]ProjectRepo + Default GitWorktrees + TaskFileSHA string +} + +func (w PerProjectGitWorktrees) Spec(t domain.Task) (string, string, bool) { + g := w.Default + if p, ok := w.Projects[t.Project]; ok && p.Repo != "" && p.WorktreeRoot != "" { + g = GitWorktrees{Root: p.WorktreeRoot, Repo: p.Repo} + } + return g.Spec(t) +} + +func (w PerProjectGitWorktrees) Create(ctx context.Context, t domain.Task) (string, error) { + g := w.Default + if p, ok := w.Projects[t.Project]; ok && p.Repo != "" && p.WorktreeRoot != "" { + g = GitWorktrees{Root: p.WorktreeRoot, Repo: p.Repo, TaskFileSHA: w.TaskFileSHA} + } + if g.TaskFileSHA == "" { + g.TaskFileSHA = w.TaskFileSHA + } + return g.Create(ctx, t) +} + +func (w PerProjectGitWorktrees) Remove(ctx context.Context, t domain.Task, path string) error { + g := w.Default + if p, ok := w.Projects[t.Project]; ok && p.Repo != "" && p.WorktreeRoot != "" { + g = GitWorktrees{Root: p.WorktreeRoot, Repo: p.Repo} + } + return g.Remove(ctx, t, path) +} + type AdapterFactory struct{ Herdrs map[string]herdr.Adapter } func (f AdapterFactory) Adapter(id string) (herdr.Adapter, error) { @@ -90,6 +158,21 @@ type MonitorHealth struct { LastRun time.Time `json:"last_run"` LastError string `json:"last_error,omitempty"` Expired int `json:"expired"` + // TurnBoundaryDegraded counts rotation ticks where Face B (spec §5.2, + // §5.3 — "the Stop-hook/Face-B decides rotation, not the router") could + // not be consulted, so occupancy-only thresholding is standing in. This + // must stay observable rather than a silent fallback: an operator (or + // the brief) can see when a deployment's rotation safety is degraded. + TurnBoundaryDegraded int `json:"turn_boundary_degraded"` + Sessions map[string]SessionHealth `json:"sessions,omitempty"` +} + +type SessionHealth struct { + Status string `json:"status,omitempty"` + WaitingForApproval bool `json:"waiting_for_approval"` + Blocker string `json:"blocker,omitempty"` + UpdatedAt time.Time `json:"updated_at"` + LastError string `json:"last_error,omitempty"` } func (c *Coordinator) MonitorHealth() MonitorHealth { @@ -110,6 +193,58 @@ func (c *Coordinator) setMonitorHealth(err error, expired int) { } } +func (c *Coordinator) recordTurnBoundaryDegraded() { + c.healthMu.Lock() + defer c.healthMu.Unlock() + c.health.TurnBoundaryDegraded++ +} + +func waitingForApproval(status string) bool { + s := strings.ToLower(strings.ReplaceAll(strings.ReplaceAll(status, "-", "_"), " ", "_")) + return s == "waiting_for_approval" || s == "awaiting_approval" || s == "approval_required" +} + +func (c *Coordinator) refreshSessionHealth(ctx context.Context) { + c.loadSessions() + c.mu.Lock() + sessions := make(map[string]herdr.Session, len(c.sessions)) + for id, s := range c.sessions { + sessions[id] = s + } + c.mu.Unlock() + c.healthMu.Lock() + if c.health.Sessions == nil { + c.health.Sessions = map[string]SessionHealth{} + } + c.healthMu.Unlock() + for taskID, session := range sessions { + adapterID := session.HerdrID + if adapterID == "" { + if task, ok := c.Store.Task(taskID); ok && task.Lease != nil { + adapterID = task.Lease.HarnessID + } + } + a, err := c.Adapters.Adapter(adapterID) + if err != nil { + continue + } + p, ok := a.(herdr.AgentStatus) + if !ok { + continue + } + status, err := p.AgentStatus(ctx, session) + h := SessionHealth{Status: status, WaitingForApproval: waitingForApproval(status), UpdatedAt: time.Now().UTC()} + if err != nil { + h.LastError = err.Error() + } else if blocker, ok := a.(herdr.AgentBlocker); ok && strings.EqualFold(status, "blocked") { + h.Blocker, _ = blocker.AgentBlocker(ctx, session) + } + c.healthMu.Lock() + c.health.Sessions[taskID] = h + c.healthMu.Unlock() + } +} + func (c *Coordinator) loadSessions() { c.mu.Lock() defer c.mu.Unlock() @@ -148,7 +283,6 @@ func (c *Coordinator) saveSessionsLocked() error { func (c *Coordinator) Reconcile(ctx context.Context) error { c.loadSessions() c.mu.Lock() - defer c.mu.Unlock() for taskID, session := range c.sessions { t, ok := c.Store.Task(taskID) if ok && t.State == domain.StateLeased { @@ -159,7 +293,16 @@ func (c *Coordinator) Reconcile(ctx context.Context) error { } delete(c.sessions, taskID) } - return c.saveSessionsLocked() + err := c.saveSessionsLocked() + c.mu.Unlock() + if err != nil { + return err + } + // Session health is derived state. Rebuild it immediately from the + // durable session mappings so a restart does not hide an outstanding + // approval until the first periodic monitor tick. + c.refreshSessionHealth(ctx) + return nil } // Monitor performs conservative hard-threshold rotation. The adapter owns @@ -183,6 +326,8 @@ func (c *Coordinator) Monitor(ctx context.Context, hard float64, interval time.D c.healthMu.Unlock() return ctx.Err() case <-t.C: + c.refreshSessionHealth(ctx) + c.cleanupCompleted(ctx) expired, err := c.expire(ctx) c.setMonitorHealth(err, len(expired)) if err != nil { @@ -193,6 +338,34 @@ func (c *Coordinator) Monitor(ctx context.Context, hard float64, interval time.D } } +func (c *Coordinator) cleanupCompleted(ctx context.Context) { + cleaner, ok := c.Worktrees.(WorktreeCleaner) + if !ok { + return + } + c.loadSessions() + c.mu.Lock() + defer c.mu.Unlock() + changed := false + for taskID, session := range c.sessions { + t, exists := c.Store.Task(taskID) + if !exists || t.State != domain.StateCompleted { + continue + } + if err := cleaner.Remove(ctx, t, session.Worktree); err != nil { + c.healthMu.Lock() + c.health.LastError = "worktree cleanup: " + err.Error() + c.healthMu.Unlock() + continue + } + delete(c.sessions, taskID) + changed = true + } + if changed { + _ = c.saveSessionsLocked() + } +} + func (c *Coordinator) expire(ctx context.Context) ([]domain.Event, error) { // pane.exited is the low-latency path; lease expiry below remains the // authoritative backstop when herdr misses an exit notification. @@ -204,7 +377,7 @@ func (c *Coordinator) expire(ctx context.Context) ([]domain.Event, error) { if p, ok := a.(herdr.PaneExit); ok { if exited, ee := p.PaneExited(ctx, s); ee == nil && exited { b, _ := json.Marshal(map[string]string{"reason": "pane_exited", "harness_id": s.Harness}) - _ = c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: t.Version + 1, Payload: b}) + _ = c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: t.Version + 1, Payload: b, Surface: string(authz.System)}) } } } @@ -257,11 +430,25 @@ func (c *Coordinator) rotate(ctx context.Context, hard float64) { if err != nil || (occupancy < hard && reason == "threshold") { continue } + // Face B is treated as required, not best-effort (spec §5.2/§5.3): + // an adapter that supports the turn-boundary probe but fails to + // answer it blocks this tick's release rather than silently + // proceeding as if mid-turn interruption were safe. Only an + // adapter that genuinely does not implement TurnBoundary at all + // falls back to occupancy-only thresholding, and that fallback is + // recorded so it is observable (MonitorHealth.TurnBoundaryDegraded) + // instead of invisible. if boundary, ok := a.(herdr.TurnBoundary); ok { atBoundary, boundaryErr := boundary.AtTurnBoundary(ctx, session) - if boundaryErr == nil && !atBoundary { + if boundaryErr != nil { + c.recordTurnBoundaryDegraded() continue } + if !atBoundary { + continue + } + } else { + c.recordTurnBoundaryDegraded() } ref, err := a.Release(ctx, session) if err != nil { @@ -270,8 +457,16 @@ func (c *Coordinator) rotate(ctx context.Context, hard float64) { if ref == "" { continue } - b, _ := json.Marshal(map[string]string{"handoff_ref": ref, "reason": reason}) - e := domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: task.Version + 1, Payload: b} + anchorSHA, err := herdr.HeadSHA(session.Worktree) + if err != nil { + // Cannot certify the anchor: do not release with an invalid + // TaskReleased payload (it would fail validation and strand + // the lease/session). Leave the lease intact for the next + // tick or TTL expiry to reclaim. + continue + } + b, _ := json.Marshal(map[string]string{"handoff_ref": ref, "reason": reason, "anchor_sha": anchorSHA}) + e := domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: task.Version + 1, Payload: b, Surface: string(authz.System)} if c.Store.Append(e) == nil { c.mu.Lock() delete(c.sessions, taskID) @@ -300,14 +495,27 @@ func (c *Coordinator) Start(ctx context.Context, e domain.Event) error { if err := json.Unmarshal(e.Payload, &p); err != nil || p.HarnessID == "" { return fmt.Errorf("orchestrator: invalid lease") } - w, err := c.Worktrees.Create(ctx, t) - if err != nil { - return c.block(t, "worktree: "+err.Error()) - } a, err := c.Adapters.Adapter(p.HarnessID) if err != nil { return c.block(t, "adapter: "+err.Error()) } + var w string + if creator, ok := a.(herdr.WorktreeCreator); ok { + planner, planned := c.Worktrees.(WorktreeSpec) + if !planned { + return c.block(t, "worktree: repository specification unavailable") + } + repo, root, valid := planner.Spec(t) + if !valid { + return c.block(t, "worktree: repository and root required") + } + w, err = creator.CreateWorktree(ctx, repo, root, t.ID) + } else { + w, err = c.Worktrees.Create(ctx, t) + } + if err != nil { + return c.block(t, "worktree: "+err.Error()) + } s, err := a.Lease(ctx, t.ID, w) if err != nil { return c.block(t, "lease: "+err.Error()) @@ -318,6 +526,7 @@ func (c *Coordinator) Start(ctx context.Context, e domain.Event) error { return c.block(t, "bootstrap: "+err.Error()) } } + s.HerdrID = p.HarnessID c.mu.Lock() if c.sessions == nil { c.sessions = map[string]herdr.Session{} @@ -325,12 +534,18 @@ func (c *Coordinator) Start(ctx context.Context, e domain.Event) error { c.sessions[t.ID] = s err = c.saveSessionsLocked() c.mu.Unlock() + c.healthMu.Lock() + if c.health.Sessions == nil { + c.health.Sessions = map[string]SessionHealth{} + } + c.health.Sessions[t.ID] = SessionHealth{Status: "running", UpdatedAt: time.Now().UTC()} + c.healthMu.Unlock() return err } func (c *Coordinator) block(t domain.Task, reason string) error { b, _ := json.Marshal(map[string]string{"blocker": reason}) - return c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskBlocked", TaskID: t.ID, Version: t.Version + 1, Payload: b}) + return c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskBlocked", TaskID: t.ID, Version: t.Version + 1, Payload: b, Surface: string(authz.System)}) } func (c *Coordinator) Session(taskID string) (herdr.Session, bool) { @@ -340,3 +555,25 @@ func (c *Coordinator) Session(taskID string) (herdr.Session, bool) { s, ok := c.sessions[taskID] return s, ok } + +func (c *Coordinator) Capture(ctx context.Context, taskID, source string) (string, error) { + s, ok := c.Session(taskID) + if !ok { + return "", domain.ErrNotFound + } + id := s.HerdrID + if id == "" { + if t, ok := c.Store.Task(taskID); ok && t.Lease != nil { + id = t.Lease.HarnessID + } + } + a, err := c.Adapters.Adapter(id) + if err != nil { + return "", err + } + p, ok := a.(herdr.PaneCapture) + if !ok { + return "", fmt.Errorf("pane capture unsupported") + } + return p.PaneCapture(ctx, s, source) +} diff --git a/internal/orchestrator/rotation_test.go b/internal/orchestrator/rotation_test.go new file mode 100644 index 0000000..b2fc42f --- /dev/null +++ b/internal/orchestrator/rotation_test.go @@ -0,0 +1,267 @@ +package orchestrator_test + +import ( + "context" + "encoding/json" + "errors" + "orchestra/internal/authz" + "orchestra/internal/domain" + "orchestra/internal/herdr" + "orchestra/internal/orchestrator" + "orchestra/internal/store" + "os/exec" + "testing" + "time" +) + +type fakeAdapter struct { + occupancy float64 + boundary bool + ref string + releases int +} + +func (a *fakeAdapter) Lease(_ context.Context, _ string, worktree string) (herdr.Session, error) { + return herdr.Session{Harness: "h1", PaneID: "pane-1", Worktree: worktree}, nil +} +func (a *fakeAdapter) Bootstrap(context.Context, herdr.Session, string) error { return nil } +func (a *fakeAdapter) Release(context.Context, herdr.Session) (string, error) { + a.releases++ + return a.ref, nil +} +func (a *fakeAdapter) Kill(context.Context, herdr.Session) error { return nil } +func (a *fakeAdapter) Occupancy(herdr.Session) (float64, error) { return a.occupancy, nil } +func (a *fakeAdapter) AtTurnBoundary(context.Context, herdr.Session) (bool, error) { + return a.boundary, nil +} + +type worktrees struct{ path string } + +func (w worktrees) Create(context.Context, domain.Task) (string, error) { return w.path, nil } + +type adapters struct{ a herdr.Adapter } + +func (a adapters) Adapter(string) (herdr.Adapter, error) { return a.a, nil } + +func run(t *testing.T, dir string, args ...string) { + t.Helper() + cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("git %v: %v: %s", args, err, out) + } +} + +// TestRotationEmitsValidReleaseWithAnchorSHA guards the highest-priority spec +// defect noted in progress.md: automated rotation must emit a TaskReleased +// event that satisfies domain.ValidatePayload (handoff_ref + anchor_sha), not +// a payload missing anchor_sha that silently fails to append. +func TestRotationEmitsValidReleaseWithAnchorSHA(t *testing.T) { + repo := t.TempDir() + run(t, repo, "init") + run(t, repo, "config", "user.email", "t@t") + run(t, repo, "config", "user.name", "t") + run(t, repo, "commit", "--allow-empty", "-m", "init") + head, err := herdr.HeadSHA(repo) + if err != nil { + t.Fatal(err) + } + + s, err := store.Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskCreated", TaskID: "t1", Surface: string(authz.System), Payload: mustJSON(map[string]any{ + "source": "jsonl", "external_id": "1", "project": "p", + })}); err != nil { + t.Fatal(err) + } + task := s.Tasks()[0] + ref, err := s.PutArtifact([]byte("handoff")) + if err != nil { + t.Fatal(err) + } + a := &fakeAdapter{occupancy: .95, boundary: true, ref: ref} + c := &orchestrator.Coordinator{Store: s, Worktrees: worktrees{path: repo}, Adapters: adapters{a}, StatePath: t.TempDir() + "/sessions.json"} + + leaseEvt, err := s.Lease(task.ID, "h1", time.Minute) + if err != nil { + t.Fatal(err) + } + if err := c.Start(context.Background(), leaseEvt); err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + go c.Monitor(ctx, .8, time.Millisecond) + + deadline := time.Now().Add(time.Second) + for time.Now().Before(deadline) { + if got, ok := s.Task(task.ID); ok && got.State == domain.StateQueued { + break + } + time.Sleep(time.Millisecond) + } + got, ok := s.Task(task.ID) + if !ok || got.State != domain.StateQueued { + t.Fatalf("rotation did not complete: state=%v ok=%v", got.State, ok) + } + if a.releases == 0 { + t.Fatalf("adapter Release was never invoked") + } + + // Walk raw events to confirm the coordinator itself wrote a valid + // TaskReleased payload with anchor_sha == the worktree's real HEAD. + found := false + for _, e := range s.Events(0) { + if e.TaskID != task.ID || e.Type != "TaskReleased" { + continue + } + var p map[string]any + if err := json.Unmarshal(e.Payload, &p); err != nil { + t.Fatal(err) + } + if err := domain.ValidatePayload("TaskReleased", p); err != nil { + t.Fatalf("coordinator emitted invalid TaskReleased: %v (%v)", err, p) + } + if p["anchor_sha"] != head { + t.Fatalf("anchor_sha=%v want=%s", p["anchor_sha"], head) + } + found = true + } + if !found { + t.Fatal("coordinator never emitted a TaskReleased event") + } +} + +func mustJSON(v any) []byte { b, _ := json.Marshal(v); return b } + +// erroringBoundaryAdapter supports Face B but its probe always fails — this +// must block release (never silently treat an unanswerable boundary check +// as safe to interrupt), unlike an adapter that doesn't implement the +// interface at all. +type erroringBoundaryAdapter struct{ fakeAdapter } + +func (a *erroringBoundaryAdapter) AtTurnBoundary(context.Context, herdr.Session) (bool, error) { + return false, errors.New("pane.status unsupported") +} + +// noBoundaryAdapter never implements herdr.TurnBoundary at all, exercising +// the genuine occupancy-only degraded fallback. +type noBoundaryAdapter struct { + occupancy float64 + ref string + releases int +} + +func (a *noBoundaryAdapter) Lease(_ context.Context, _ string, worktree string) (herdr.Session, error) { + return herdr.Session{Harness: "h1", PaneID: "pane-1", Worktree: worktree}, nil +} +func (a *noBoundaryAdapter) Bootstrap(context.Context, herdr.Session, string) error { return nil } +func (a *noBoundaryAdapter) Release(context.Context, herdr.Session) (string, error) { + a.releases++ + return a.ref, nil +} +func (a *noBoundaryAdapter) Kill(context.Context, herdr.Session) error { return nil } +func (a *noBoundaryAdapter) Occupancy(herdr.Session) (float64, error) { return a.occupancy, nil } + +func setupRotationTask(t *testing.T, repo string) (*store.Store, string, domain.Task, string) { + t.Helper() + run(t, repo, "init") + run(t, repo, "config", "user.email", "t@t") + run(t, repo, "config", "user.name", "t") + run(t, repo, "commit", "--allow-empty", "-m", "init") + head, err := herdr.HeadSHA(repo) + if err != nil { + t.Fatal(err) + } + s, err := store.Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskCreated", TaskID: "t1", Surface: string(authz.System), Payload: mustJSON(map[string]any{ + "source": "jsonl", "external_id": "1", "project": "p", + })}); err != nil { + t.Fatal(err) + } + task := s.Tasks()[0] + ref, err := s.PutArtifact([]byte("handoff")) + if err != nil { + t.Fatal(err) + } + return s, head, task, ref +} + +// TestTurnBoundaryErrorBlocksRelease proves an adapter that implements Face B +// but cannot currently answer it (a transient herdr error) never falls +// through to an unconfirmed release — spec §5.2/§5.3 treats the boundary +// check as required, not best-effort. +func TestTurnBoundaryErrorBlocksRelease(t *testing.T) { + repo := t.TempDir() + s, _, task, ref := setupRotationTask(t, repo) + a := &erroringBoundaryAdapter{fakeAdapter{occupancy: .95, boundary: true, ref: ref}} + c := &orchestrator.Coordinator{Store: s, Worktrees: worktrees{path: repo}, Adapters: adapters{a}, StatePath: t.TempDir() + "/sessions.json"} + + leaseEvt, err := s.Lease(task.ID, "h1", time.Minute) + if err != nil { + t.Fatal(err) + } + if err := c.Start(context.Background(), leaseEvt); err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + go c.Monitor(ctx, .8, time.Millisecond) + + time.Sleep(50 * time.Millisecond) + got, _ := s.Task(task.ID) + if got.State != domain.StateLeased { + t.Fatalf("release proceeded despite an unanswerable turn-boundary check: state=%s", got.State) + } + if a.releases != 0 { + t.Fatalf("adapter.Release was called despite the boundary error, releases=%d", a.releases) + } + if c.MonitorHealth().TurnBoundaryDegraded == 0 { + t.Fatal("turn-boundary degradation was not recorded") + } +} + +// TestNoTurnBoundarySupportDegradesVisibly proves an adapter that never +// implements Face B still falls back to occupancy-only thresholding (so +// existing deployments keep working) but the degradation is observable via +// MonitorHealth, not silent. +func TestNoTurnBoundarySupportDegradesVisibly(t *testing.T) { + repo := t.TempDir() + s, head, task, ref := setupRotationTask(t, repo) + a := &noBoundaryAdapter{occupancy: .95, ref: ref} + c := &orchestrator.Coordinator{Store: s, Worktrees: worktrees{path: repo}, Adapters: adapters{a}, StatePath: t.TempDir() + "/sessions.json"} + + leaseEvt, err := s.Lease(task.ID, "h1", time.Minute) + if err != nil { + t.Fatal(err) + } + if err := c.Start(context.Background(), leaseEvt); err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + go c.Monitor(ctx, .8, time.Millisecond) + + deadline := time.Now().Add(time.Second) + for time.Now().Before(deadline) { + if got, ok := s.Task(task.ID); ok && got.State == domain.StateQueued { + break + } + time.Sleep(time.Millisecond) + } + got, ok := s.Task(task.ID) + if !ok || got.State != domain.StateQueued { + t.Fatalf("rotation did not complete without Face B support: state=%v ok=%v", got.State, ok) + } + _ = head + if c.MonitorHealth().TurnBoundaryDegraded == 0 { + t.Fatal("missing Face B support was not recorded as degraded") + } +} diff --git a/internal/orchestrator/worktrees_test.go b/internal/orchestrator/worktrees_test.go new file mode 100644 index 0000000..50fb771 --- /dev/null +++ b/internal/orchestrator/worktrees_test.go @@ -0,0 +1,63 @@ +package orchestrator_test + +import ( + "context" + "orchestra/internal/domain" + "orchestra/internal/orchestrator" + "os" + "os/exec" + "path/filepath" + "testing" +) + +func initRepo(t *testing.T, dir string) { + t.Helper() + run := func(args ...string) { + cmd := exec.Command("git", args...) + cmd.Dir = dir + cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@t") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("git %v: %v: %s", args, err, out) + } + } + if err := os.MkdirAll(dir, 0755); err != nil { + t.Fatal(err) + } + run("init") + if err := os.WriteFile(filepath.Join(dir, "README"), []byte("x"), 0644); err != nil { + t.Fatal(err) + } + run("add", "README") + run("commit", "-m", "init") +} + +func TestPerProjectGitWorktreesResolvesByProject(t *testing.T) { + base := t.TempDir() + repoA := filepath.Join(base, "repo-a") + repoB := filepath.Join(base, "repo-b") + initRepo(t, repoA) + initRepo(t, repoB) + + w := orchestrator.PerProjectGitWorktrees{ + Projects: map[string]orchestrator.ProjectRepo{ + "proj-a": {Repo: repoA, WorktreeRoot: filepath.Join(base, "wt-a")}, + }, + Default: orchestrator.GitWorktrees{Root: filepath.Join(base, "wt-default"), Repo: repoB}, + } + + pathA, err := w.Create(context.Background(), domain.Task{ID: "t1", Project: "proj-a"}) + if err != nil { + t.Fatalf("create for proj-a: %v", err) + } + if filepath.Dir(pathA) != filepath.Join(base, "wt-a") { + t.Fatalf("expected proj-a worktree under wt-a, got %s", pathA) + } + + pathDefault, err := w.Create(context.Background(), domain.Task{ID: "t2", Project: "unconfigured-project"}) + if err != nil { + t.Fatalf("create for unconfigured project: %v", err) + } + if filepath.Dir(pathDefault) != filepath.Join(base, "wt-default") { + t.Fatalf("expected unconfigured project to use default worktree root, got %s", pathDefault) + } +} diff --git a/internal/provider/gitea_test.go b/internal/provider/gitea_test.go new file mode 100644 index 0000000..495e004 --- /dev/null +++ b/internal/provider/gitea_test.go @@ -0,0 +1,136 @@ +package provider + +import ( + "crypto/hmac" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "net/http" + "net/http/httptest" + "orchestra/internal/domain" + "os" + "path/filepath" + "testing" +) + +func TestGiteaSourceNameDefaultsToRepo(t *testing.T) { + if got := (Gitea{Repo: "orchestra"}).SourceName(); got != "gitea" { + t.Fatalf("expected legacy unnamespaced source, got %q", got) + } + if got := (Gitea{Repo: "orchestra", Project: "correx"}).SourceName(); got != "gitea:correx" { + t.Fatalf("expected namespaced source, got %q", got) + } +} + +func TestGiteaIngestWebhookTagsProject(t *testing.T) { + g := Gitea{Owner: "kami", Repo: "correx-repo", Project: "correx", WebhookSecret: "s3cret"} + body, _ := json.Marshal(map[string]any{ + "action": "opened", + "issue": map[string]any{"number": 42, "title": "fix thing", "body": "", "state": "open"}, + }) + mac := hmac.New(sha256.New, []byte("s3cret")) + mac.Write(body) + sig := hex.EncodeToString(mac.Sum(nil)) + + sk := &sink{} + if err := g.IngestWebhook(body, sig, sk); err != nil { + t.Fatalf("ingest: %v", err) + } + if len(sk.events) != 1 { + t.Fatalf("expected 1 event, got %d", len(sk.events)) + } + var p struct { + Project string `json:"project"` + Source string `json:"source"` + ExternalID string `json:"external_id"` + } + if err := json.Unmarshal(sk.events[0].Payload, &p); err != nil { + t.Fatalf("unmarshal payload: %v", err) + } + if p.Project != "correx" || p.Source != "gitea:correx" || p.ExternalID != "42" { + t.Fatalf("unexpected payload: %+v", p) + } +} + +func TestGiteaIngestWebhookRejectsBadSignature(t *testing.T) { + g := Gitea{Owner: "kami", Repo: "r", Project: "p", WebhookSecret: "s3cret"} + if err := g.IngestWebhook([]byte(`{"action":"opened","issue":{"number":1}}`), "wrong", &sink{}); err == nil { + t.Fatal("expected signature rejection") + } +} + +func TestMultiGiteaReflectDispatchesByTaskSource(t *testing.T) { + var hitCorrex, hitMaven bool + correxSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hitCorrex = true + w.WriteHeader(200) + })) + defer correxSrv.Close() + mavenSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hitMaven = true + w.WriteHeader(200) + })) + defer mavenSrv.Close() + + m := MultiGitea{Sources: map[string]Gitea{ + "gitea:correx": {BaseURL: correxSrv.URL, Owner: "kami", Repo: "correx-repo", Project: "correx"}, + "gitea:maven": {BaseURL: mavenSrv.URL, Owner: "kami", Repo: "maven-repo", Project: "maven"}, + }} + + if err := m.ReflectTask(domain.Task{Source: "gitea:correx", ExternalID: "7"}, domain.Event{Type: "TaskCompleted"}); err != nil { + t.Fatalf("reflect to correx: %v", err) + } + if !hitCorrex || hitMaven { + t.Fatalf("expected only correx server hit, got correx=%v maven=%v", hitCorrex, hitMaven) + } + + hitCorrex, hitMaven = false, false + if err := m.ReflectTask(domain.Task{Source: "gitea:maven", ExternalID: "3"}, domain.Event{Type: "TaskFailed"}); err != nil { + t.Fatalf("reflect to maven: %v", err) + } + if hitCorrex || !hitMaven { + t.Fatalf("expected only maven server hit, got correx=%v maven=%v", hitCorrex, hitMaven) + } + + // A task from a non-Gitea source (or an unregistered Gitea project) must + // be a silent no-op, not an error. + if err := m.ReflectTask(domain.Task{Source: "jsonl"}, domain.Event{Type: "TaskCompleted"}); err != nil { + t.Fatalf("unmatched source should no-op, got %v", err) + } +} + +func TestLoadGiteaConfigsValidatesAndRejectsDuplicates(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "gitea.json") + good := `[ + {"project":"correx","base_url":"https://gitea.internal","owner":"kami","repo":"correx"}, + {"project":"maven","base_url":"https://gitea.internal","owner":"kami","repo":"maven"} + ]` + if err := os.WriteFile(path, []byte(good), 0644); err != nil { + t.Fatal(err) + } + cfgs, err := LoadGiteaConfigs(path) + if err != nil || len(cfgs) != 2 { + t.Fatalf("cfgs=%d err=%v", len(cfgs), err) + } + + dupPath := filepath.Join(dir, "dup.json") + dup := `[ + {"project":"correx","base_url":"https://gitea.internal","owner":"kami","repo":"correx"}, + {"project":"correx","base_url":"https://gitea.internal","owner":"kami","repo":"other"} + ]` + if err := os.WriteFile(dupPath, []byte(dup), 0644); err != nil { + t.Fatal(err) + } + if _, err := LoadGiteaConfigs(dupPath); err == nil { + t.Fatal("expected duplicate project rejection") + } + + incompletePath := filepath.Join(dir, "incomplete.json") + if err := os.WriteFile(incompletePath, []byte(`[{"project":"x"}]`), 0644); err != nil { + t.Fatal(err) + } + if _, err := LoadGiteaConfigs(incompletePath); err == nil { + t.Fatal("expected missing-field rejection") + } +} diff --git a/internal/provider/provider.go b/internal/provider/provider.go index c2e6557..dc5ec56 100644 --- a/internal/provider/provider.go +++ b/internal/provider/provider.go @@ -18,6 +18,7 @@ import ( "sync" "time" + "orchestra/internal/authz" "orchestra/internal/domain" ) @@ -136,7 +137,7 @@ func (j JSONL) Ingest(r io.Reader, sink Sink) (int, error) { return count, fmt.Errorf("line %d: %w", line, err) } b, _ := json.Marshal(p) - if err := sink.Append(domain.Event{ID: domain.NewID(), Type: "TaskCreated", TaskID: domain.NewID(), Version: 1, Payload: b}); err != nil { + if err := sink.Append(domain.Event{ID: domain.NewID(), Type: "TaskCreated", TaskID: domain.NewID(), Version: 1, Payload: b, Surface: string(authz.System)}); err != nil { return count, fmt.Errorf("line %d: %w", line, err) } count++ @@ -197,7 +198,78 @@ func (w JSONLWatcher) Run(ctx context.Context, sink Sink) error { type Gitea struct { BaseURL, Token, WebhookSecret, Owner, Repo string - Client *http.Client + // Project, if set, is the orchestra project id ingested tasks are + // tagged with (registry.Project.ID) and the key this source is + // dispatched under in MultiGitea. Deployments with a single Gitea repo + // may leave it empty, in which case Repo is used as both — preserving + // the historical single-source behavior. + Project string + Client *http.Client +} + +// sourceName is the provider "source" every ingested TaskCreated carries, +// and the (source,external_id) idempotency/reflection key. It is namespaced +// per project so issue numbers from two different Gitea repos never +// collide in the dedup key, and so MultiGitea can route a TaskCompleted's +// reflection back to the correct repo. +func (g Gitea) sourceName() string { return g.SourceName() } + +// SourceName is the exported form of sourceName, for callers (e.g. +// cmd/orchestra) that build a MultiGitea{Sources: ...} map. +func (g Gitea) SourceName() string { + if g.Project == "" { + return "gitea" + } + return "gitea:" + g.Project +} + +// GiteaSourceConfig describes one Gitea repo to ingest from/reflect to. +// Load a list of these from JSON (ORCHESTRA_GITEA_CONFIG) to run more than +// one Gitea-backed project side by side — each project may have its own +// repo, owner, and credentials. +type GiteaSourceConfig struct { + Project string `json:"project"` + BaseURL string `json:"base_url"` + Owner string `json:"owner"` + Repo string `json:"repo"` + Token string `json:"token"` + WebhookSecret string `json:"webhook_secret"` +} + +// LoadGiteaConfigs reads a JSON array of GiteaSourceConfig from path. +func LoadGiteaConfigs(path string) ([]GiteaSourceConfig, error) { + b, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var out []GiteaSourceConfig + if err := json.Unmarshal(b, &out); err != nil { + return nil, fmt.Errorf("gitea config: %w", err) + } + seen := map[string]bool{} + for _, c := range out { + if c.Project == "" || c.BaseURL == "" || c.Owner == "" || c.Repo == "" { + return nil, fmt.Errorf("gitea config: project, base_url, owner, and repo are required (got %+v)", c) + } + if seen[c.Project] { + return nil, fmt.Errorf("gitea config: duplicate project %q", c.Project) + } + seen[c.Project] = true + } + return out, nil +} + +// MultiGitea dispatches TaskReflector reflection to whichever Gitea source +// ingested the task, keyed by Gitea.sourceName(). This lets several Gitea +// repos (one per project) share a single ReflectingSink. +type MultiGitea struct{ Sources map[string]Gitea } + +func (m MultiGitea) ReflectTask(task domain.Task, e domain.Event) error { + g, ok := m.Sources[task.Source] + if !ok { + return nil + } + return g.ReflectTask(task, e) } type giteaIssue struct { Number int `json:"number"` @@ -229,7 +301,7 @@ func (g Gitea) event(issue giteaIssue, source, project string) domain.Event { } p := map[string]any{"source": source, "external_id": strconv.Itoa(issue.Number), "project": project, "title": issue.Title, "capability": caps} b, _ := json.Marshal(p) - return domain.Event{ID: domain.NewID(), TaskID: domain.NewID(), Type: "TaskCreated", Version: 1, Payload: b} + return domain.Event{ID: domain.NewID(), TaskID: domain.NewID(), Type: "TaskCreated", Version: 1, Payload: b, Surface: string(authz.System)} } func (g Gitea) IngestWebhook(body []byte, signature string, sink Sink) error { if !validSignature(body, signature, g.WebhookSecret) { @@ -242,11 +314,14 @@ func (g Gitea) IngestWebhook(body []byte, signature string, sink Sink) error { if h.Action == "closed" || h.Action == "deleted" { return nil } - project := g.Repo - if h.Repository.FullName != "" { - project = h.Repository.FullName + project := g.Project + if project == "" { + project = g.Repo + if h.Repository.FullName != "" { + project = h.Repository.FullName + } } - return sink.Append(g.event(h.Issue, "gitea", project)) + return sink.Append(g.event(h.Issue, g.sourceName(), project)) } func validSignature(body []byte, got, secret string) bool { if secret == "" || got == "" { @@ -299,8 +374,12 @@ func (g Gitea) Poll(ctx context.Context, sink Sink) (int, error) { if err = json.NewDecoder(resp.Body).Decode(&issues); err != nil { return 0, err } + project := g.Project + if project == "" { + project = g.Repo + } for _, i := range issues { - if err := sink.Append(g.event(i, "gitea", g.Repo)); err != nil { + if err := sink.Append(g.event(i, g.sourceName(), project)); err != nil { return 0, err } } diff --git a/internal/registry/registry.go b/internal/registry/registry.go index 06cefd7..6b41e41 100644 --- a/internal/registry/registry.go +++ b/internal/registry/registry.go @@ -21,6 +21,13 @@ var ( type Project struct { ID string `json:"id"` MachineAffinity []string `json:"machine_affinity"` + // Repo and WorktreeRoot let each project resolve its own git checkout + // (spec §2.2 — a project is first-class; nothing about the model implies + // a single shared repo across all projects). Both optional: a project + // that omits them falls back to whatever global default the deployment + // wires (single-repo deployments keep working unchanged). + Repo string `json:"repo,omitempty"` + WorktreeRoot string `json:"worktree_root,omitempty"` } type Machine struct { ID string `json:"id"` @@ -34,8 +41,17 @@ type Herdr struct { Protocol string `json:"protocol,omitempty"` Capabilities []string `json:"capabilities"` Concurrency int `json:"concurrency"` - QuotaLimit float64 `json:"quota_limit,omitempty"` + // QuotaLimit is deprecated in favor of QuotaLimit5h/QuotaLimitWeekly; if + // set and QuotaLimit5h is not, it is treated as the weekly limit only + // (its historical meaning), to avoid silently inventing a 5h cap for + // existing configuration. + QuotaLimit float64 `json:"quota_limit,omitempty"` + QuotaLimit5h float64 `json:"quota_limit_5h,omitempty"` + QuotaLimitWeekly float64 `json:"quota_limit_weekly,omitempty"` } + +const defaultHerdrPort = "9245" + type Config struct { Projects []Project `json:"projects"` Machines []Machine `json:"machines"` @@ -53,12 +69,58 @@ func Load(path string) (Registry, error) { return Registry{}, err } var c Config - if err = json.Unmarshal(b, &c); err != nil { + if err = json.Unmarshal(stripJSONComments(b), &c); err != nil { return Registry{}, fmt.Errorf("registry config: %w", err) } return New(c) } +// stripJSONComments removes // line comments and /* */ block comments from +// JSONC input, leaving valid JSON. Comment markers inside string literals +// (respecting backslash escapes) are left untouched. This lets deployments +// annotate config.json in place instead of keeping a separate undocumented +// copy (see deploy/config.example.jsonc). +func stripJSONComments(b []byte) []byte { + out := make([]byte, 0, len(b)) + inString, escaped, inLineComment, inBlockComment := false, false, false, false + for i := 0; i < len(b); i++ { + c := b[i] + switch { + case inLineComment: + if c == '\n' { + inLineComment = false + out = append(out, c) + } + case inBlockComment: + if c == '*' && i+1 < len(b) && b[i+1] == '/' { + inBlockComment = false + i++ + } + case inString: + out = append(out, c) + if escaped { + escaped = false + } else if c == '\\' { + escaped = true + } else if c == '"' { + inString = false + } + case c == '"': + inString = true + out = append(out, c) + case c == '/' && i+1 < len(b) && b[i+1] == '/': + inLineComment = true + i++ + case c == '/' && i+1 < len(b) && b[i+1] == '*': + inBlockComment = true + i++ + default: + out = append(out, c) + } + } + return out +} + func New(c Config) (Registry, error) { r := Registry{map[string]Project{}, map[string]Machine{}, map[string]Herdr{}} for _, p := range c.Projects { @@ -160,6 +222,9 @@ func (r Registry) Candidates(project string, check Reachability, timeout time.Du addr := h.Address if addr == "" { addr = r.machines[h.MachineID].Address + if host, _, err := net.SplitHostPort(addr); err == nil { + addr = net.JoinHostPort(host, defaultHerdrPort) + } } if check == nil || check.Reachable(addr, timeout) { out = append(out, h) diff --git a/internal/router/contract_test.go b/internal/router/contract_test.go index d3f308d..96f63af 100644 --- a/internal/router/contract_test.go +++ b/internal/router/contract_test.go @@ -5,6 +5,7 @@ import ( "testing" "time" + "orchestra/internal/authz" "orchestra/internal/domain" "orchestra/internal/registry" "orchestra/internal/store" @@ -17,7 +18,7 @@ func TestAssignPendingUsesProjectAffinityAndCapability(t *testing.T) { } add := func(id, project string, caps []string) { b, _ := json.Marshal(map[string]any{"source": "test", "external_id": id, "project": project, "capability": caps}) - if err := s.Append(domain.Event{Type: "TaskCreated", TaskID: id, Version: 1, Payload: b}); err != nil { + if err := s.Append(domain.Event{Type: "TaskCreated", TaskID: id, Version: 1, Payload: b, Surface: string(authz.System)}); err != nil { t.Fatal(err) } } @@ -45,7 +46,7 @@ func TestAssignPendingDoesNotPreemptRunningWork(t *testing.T) { t.Fatal(err) } b, _ := json.Marshal(map[string]any{"source": "test", "external_id": "queued", "project": "p", "capability": []string{}}) - if err := s.Append(domain.Event{Type: "TaskCreated", TaskID: "queued", Version: 1, Payload: b}); err != nil { + if err := s.Append(domain.Event{Type: "TaskCreated", TaskID: "queued", Version: 1, Payload: b, Surface: string(authz.System)}); err != nil { t.Fatal(err) } r, err := registry.New(registry.Config{Projects: []registry.Project{{ID: "p", MachineAffinity: []string{"m"}}}, Machines: []registry.Machine{{ID: "m", Address: "m:1"}}, Herdrs: []registry.Herdr{{ID: "h", MachineID: "m", Concurrency: 1}}}) diff --git a/internal/router/router.go b/internal/router/router.go index b1ee185..1d7bd7f 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -4,6 +4,7 @@ package router import ( "encoding/json" "errors" + "orchestra/internal/authz" "orchestra/internal/domain" "orchestra/internal/registry" "orchestra/internal/store" @@ -17,46 +18,71 @@ type AlwaysAvailable struct{} func (AlwaysAvailable) Available(registry.Herdr) bool { return true } -// QuotaAvailability applies the conservative 80% rule to summed native -// receipts in the configured rolling window. Receipts are additive across -// rotations; a cumulative report must not replace earlier rotations. +// QuotaWindowLimits are the two independent caps the spec (§7.2) requires: +// the subscription pool's 5-hour rolling window and its weekly window. They +// are tracked and evaluated separately — a harness deep into its 5h window +// but fine on the week, or vice versa, must still be excluded. +type QuotaWindowLimits struct { + FiveHour float64 + Weekly float64 +} + +const ( + fiveHourWindow = 5 * time.Hour + weeklyWindow = 7 * 24 * time.Hour + // quotaConservativeFraction is the degrade-safe default from spec §7.2/§9 + // item 1: since no quota pool is authoritative, treat 80% reported as + // full rather than trusting the exact number. + quotaConservativeFraction = 0.8 +) + +// QuotaAvailability applies the conservative 80% rule independently to the +// 5-hour rolling window and the weekly window, per harness. Receipts are +// additive across rotations; a cumulative session total must never replace +// earlier rotations' receipts (spec §5.2.1) — summing native per-report +// `consumed` deltas is what keeps this correct across rotation. type QuotaAvailability struct { Store *store.Store - Limits map[string]float64 - Window time.Duration + Limits map[string]QuotaWindowLimits Now func() time.Time } -func (q QuotaAvailability) Available(h registry.Herdr) bool { - if q.Store == nil { - return false - } - limit, bounded := q.Limits[h.ID] - if !bounded || limit <= 0 { - return true - } - now := time.Now() - if q.Now != nil { - now = q.Now() - } - window := q.Window - if window <= 0 { - window = 7 * 24 * time.Hour - } +func (q QuotaAvailability) sumSince(harnessID string, since time.Time) float64 { var consumed float64 for _, e := range q.Store.Events(0) { - if e.Type != "QuotaReported" || e.At.Before(now.Add(-window)) { + if e.Type != "QuotaReported" || e.At.Before(since) { continue } var p struct { HarnessID string `json:"harness_id"` Consumed float64 `json:"consumed"` } - if json.Unmarshal(e.Payload, &p) == nil && p.HarnessID == h.ID && p.Consumed >= 0 { + if json.Unmarshal(e.Payload, &p) == nil && p.HarnessID == harnessID && p.Consumed >= 0 { consumed += p.Consumed } } - return consumed < limit*0.8 + return consumed +} + +func (q QuotaAvailability) Available(h registry.Herdr) bool { + if q.Store == nil { + return false + } + limits, bounded := q.Limits[h.ID] + if !bounded || (limits.FiveHour <= 0 && limits.Weekly <= 0) { + return true + } + now := time.Now() + if q.Now != nil { + now = q.Now() + } + if limits.FiveHour > 0 && q.sumSince(h.ID, now.Add(-fiveHourWindow)) >= limits.FiveHour*quotaConservativeFraction { + return false + } + if limits.Weekly > 0 && q.sumSince(h.ID, now.Add(-weeklyWindow)) >= limits.Weekly*quotaConservativeFraction { + return false + } + return true } type RetryPolicy struct { @@ -185,6 +211,6 @@ func importance(t domain.Task, now time.Time) time.Time { } func (r *Router) fail(t domain.Task) (domain.Event, error) { b, _ := json.Marshal(map[string]any{"reason": "retry_limit", "attempts": r.attempts[t.ID]}) - e := domain.Event{ID: domain.NewID(), Type: "TaskFailed", TaskID: t.ID, Version: t.Version + 1, Payload: b} + e := domain.Event{ID: domain.NewID(), Type: "TaskFailed", TaskID: t.ID, Version: t.Version + 1, Payload: b, Surface: string(authz.System)} return e, r.Store.Append(e) } diff --git a/internal/router/router_test.go b/internal/router/router_test.go index c948f26..d0e49c6 100644 --- a/internal/router/router_test.go +++ b/internal/router/router_test.go @@ -2,6 +2,7 @@ package router import ( "encoding/json" + "orchestra/internal/authz" "orchestra/internal/domain" "orchestra/internal/registry" "orchestra/internal/store" @@ -28,7 +29,7 @@ func TestAssignsByAffinityCapabilityAndConcurrency(t *testing.T) { } makeTask := func(id string) { b, _ := json.Marshal(map[string]any{"source": "test", "external_id": id, "project": "p", "capability": []string{"go"}}) - if err := s.Append(domain.Event{ID: id, TaskID: id, Type: "TaskCreated", Version: 1, Payload: b}); err != nil { + if err := s.Append(domain.Event{ID: id, TaskID: id, Type: "TaskCreated", Version: 1, Payload: b, Surface: string(authz.System)}); err != nil { t.Fatal(err) } } @@ -43,3 +44,45 @@ func TestAssignsByAffinityCapabilityAndConcurrency(t *testing.T) { t.Fatal("no task leased") } } + +// TestQuotaWindowsAreIndependent proves the 5-hour rolling window and the +// weekly window (spec §7.2, §9 item 1) are each conservative-80%-full gates +// on their own — a harness can be fine on one window and excluded by the +// other, and receipts outside a window must not count toward it. +func TestQuotaWindowsAreIndependent(t *testing.T) { + s, err := store.Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + now := time.Now().UTC() + report := func(at time.Time, consumed float64) { + p, _ := json.Marshal(map[string]any{"harness_id": "h1", "consumed": consumed}) + if err := s.Append(domain.Event{ID: domain.NewID(), Type: "QuotaReported", TaskID: "quota", Version: 1, Surface: string(authz.System), Payload: p, At: at}); err != nil { + t.Fatal(err) + } + } + + // Case 1: only weekly limit configured. A receipt older than 5h but + // within the week still counts toward the weekly gate. + report(now.Add(-6*time.Hour), 85) + weeklyOnly := QuotaAvailability{Store: s, Limits: map[string]QuotaWindowLimits{"h1": {Weekly: 100}}, Now: func() time.Time { return now }} + if weeklyOnly.Available(registry.Herdr{ID: "h1"}) { + t.Fatal("weekly window should be exhausted at 85/100 (>=80%)") + } + + // Case 2: only a 5h limit configured. The same 6h-old receipt is outside + // the 5h window and must not count. + fiveHourOnly := QuotaAvailability{Store: s, Limits: map[string]QuotaWindowLimits{"h1": {FiveHour: 100}}, Now: func() time.Time { return now }} + if !fiveHourOnly.Available(registry.Herdr{ID: "h1"}) { + t.Fatal("receipt outside the 5h window incorrectly counted against it") + } + + // Case 3: a fresh receipt inside the 5h window trips the 5h gate even + // though the weekly gate (fed by both receipts) also trips — both are + // independently enforced, and either failing excludes the harness. + report(now.Add(-time.Minute), 90) + both := QuotaAvailability{Store: s, Limits: map[string]QuotaWindowLimits{"h1": {FiveHour: 100, Weekly: 500}}, Now: func() time.Time { return now }} + if both.Available(registry.Herdr{ID: "h1"}) { + t.Fatal("5h window should be exhausted at 90/100 (>=80%) regardless of weekly headroom") + } +} diff --git a/internal/store/store.go b/internal/store/store.go index cda911c..cf3714c 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -5,6 +5,7 @@ import ( "encoding/json" "errors" "fmt" + "orchestra/internal/authz" "orchestra/internal/domain" "os" "path/filepath" @@ -45,6 +46,10 @@ func Open(dir string) (*Store, error) { s.external[t.Source+"\x00"+t.ExternalID] = t.ID } snapshotSeq = snap.Seq + // Continue event numbering after the snapshot. Without restoring this + // cursor, the first append after a restart reused sequence 1 and made + // the append-only log unreplayable. + s.seq = snapshotSeq } else if !errors.Is(readErr, os.ErrNotExist) { return nil, readErr } @@ -160,9 +165,6 @@ func (s *Store) apply(e domain.Event) error { func (s *Store) Append(e domain.Event) error { s.mu.Lock() defer s.mu.Unlock() - if err := domain.ValidateEvent(e); err != nil { - return err - } if e.At.IsZero() { e.At = time.Now().UTC() } @@ -172,6 +174,15 @@ func (s *Store) Append(e domain.Event) error { if e.SchemaVersion == 0 { e.SchemaVersion = domain.CurrentEventSchema } + if err := domain.ValidateEvent(e); err != nil { + return err + } + // Enforced once, at the append boundary, per spec §7.1/invariant 4 — every + // producer (HTTP handler, router, coordinator, provider, federation relay) + // must declare its Surface here; there is no separate in-process bypass. + if err := authz.AuthorizeEvent(authz.Surface(e.Surface), e.Type); err != nil { + return err + } if e.Type == "TaskCreated" { var p map[string]any if err := json.Unmarshal(e.Payload, &p); err != nil { @@ -327,7 +338,7 @@ func (s *Store) Lease(id, harness string, ttl time.Duration) (domain.Event, erro return domain.Event{}, domain.ErrConflict } p, _ := json.Marshal(map[string]any{"harness_id": harness, "ttl": ttl.Seconds(), "until_ns": time.Now().Add(ttl).UnixNano(), "expected_version": t.Version}) - e := domain.Event{ID: id, Type: "TaskLeased", TaskID: id, Version: t.Version + 1, Payload: p} + e := domain.Event{ID: id, Type: "TaskLeased", TaskID: id, Version: t.Version + 1, Payload: p, Surface: string(authz.System)} return e, s.Append(e) } @@ -336,7 +347,7 @@ func (s *Store) ExpireLeases(now time.Time) ([]domain.Event, error) { for _, t := range s.Tasks() { if t.State == domain.StateLeased && t.Lease != nil && !t.Lease.Until.After(now) { p, _ := json.Marshal(map[string]any{"reason": "lease_expired", "harness_id": t.Lease.HarnessID}) - e := domain.Event{ID: t.ID, Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p} + e := domain.Event{ID: t.ID, Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)} if err := s.Append(e); err != nil { return out, err } diff --git a/internal/store/store_test.go b/internal/store/store_test.go index 2588c91..2d26598 100644 --- a/internal/store/store_test.go +++ b/internal/store/store_test.go @@ -6,12 +6,13 @@ import ( "path/filepath" "testing" + "orchestra/internal/authz" "orchestra/internal/domain" ) func created(id string) domain.Event { b, _ := json.Marshal(map[string]any{"source": "jsonl", "external_id": "42", "project": "demo", "capability": []string{"mechanical"}}) - return domain.Event{ID: id, Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b} + return domain.Event{ID: id, Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b, Surface: string(authz.System)} } func TestAppendReplayAndDeduplicate(t *testing.T) { @@ -34,10 +35,10 @@ func TestAppendReplayAndDeduplicate(t *testing.T) { t.Fatal(err) } completion, _ := json.Marshal(map[string]any{"report_ref": ref, "receipt": map[string]any{"harness_id": "h", "consumed": 1}}) - if err := s.Append(domain.Event{Type: "TaskCompleted", TaskID: "task-1", Version: 2, Payload: completion}); err != nil { + if err := s.Append(domain.Event{Type: "TaskCompleted", TaskID: "task-1", Version: 2, Payload: completion, Surface: string(authz.System)}); err != nil { t.Fatal(err) } - if err := s.Append(domain.Event{Type: "TaskReleased", TaskID: "task-1", Version: 2, Payload: json.RawMessage(`{"handoff_ref":"` + ref + `","anchor_sha":"0123456789012345678901234567890123456789"}`)}); err != domain.ErrConflict { + if err := s.Append(domain.Event{Type: "TaskReleased", TaskID: "task-1", Version: 2, Surface: string(authz.System), Payload: json.RawMessage(`{"handoff_ref":"` + ref + `","anchor_sha":"0123456789012345678901234567890123456789"}`)}); err != domain.ErrConflict { t.Fatalf("expected conflict, got %v", err) } s2, err := Open(dir) @@ -85,7 +86,7 @@ func TestLifecycleEventsRequireEvidence(t *testing.T) { } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - err := domain.ValidateEvent(domain.Event{Type: tc.typ, TaskID: "task-1", Version: 1, Payload: json.RawMessage(tc.body)}) + err := domain.ValidateEvent(domain.Event{Type: tc.typ, TaskID: "task-1", Version: 1, Payload: json.RawMessage(tc.body), Surface: string(authz.System)}) if err == nil { t.Fatal("expected lifecycle evidence validation error") } @@ -93,6 +94,38 @@ func TestLifecycleEventsRequireEvidence(t *testing.T) { } } +// TestAppendEnforcesAuthorizationAtTheBus proves authorization is checked +// once at the append boundary (spec §7.1/§1.4), not only in HTTP handlers: +// a caller writing to the store directly with a notify-only surface, or with +// no declared surface at all, is rejected exactly like an HTTP request would +// be — there is no in-process bypass for the router, coordinator, or a +// provider adapter that forgets to declare who it's acting as. +func TestAppendEnforcesAuthorizationAtTheBus(t *testing.T) { + s, err := Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + b, _ := json.Marshal(map[string]any{"source": "jsonl", "external_id": "1", "project": "demo"}) + + // A notify-only surface (e.g. Telegram) must never be able to create a + // task by calling the store directly, even though it bypasses HTTP. + if err := s.Append(domain.Event{ID: "e1", Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b, Surface: string(authz.Telegram)}); err == nil { + t.Fatal("notify-only surface created a task via direct store access") + } + + // An internal producer that forgets to declare a surface is rejected, + // not silently trusted as the plane. + if err := s.Append(domain.Event{ID: "e2", Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b}); err == nil { + t.Fatal("event with no declared surface was accepted") + } + + // The plane (router/coordinator/provider) authorizes as System and + // succeeds. + if err := s.Append(domain.Event{ID: "e3", Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b, Surface: string(authz.System)}); err != nil { + t.Fatalf("system surface rejected: %v", err) + } +} + func TestExpectedVersionIsCheckedForEveryWriter(t *testing.T) { s, err := Open(t.TempDir()) if err != nil { @@ -102,7 +135,7 @@ func TestExpectedVersionIsCheckedForEveryWriter(t *testing.T) { t.Fatal(err) } p := json.RawMessage(`{"reason":"rotate","expected_version":0}`) - err = s.Append(domain.Event{Type: "TaskReleased", TaskID: "task-1", Version: 2, Payload: p}) + err = s.Append(domain.Event{Type: "TaskReleased", TaskID: "task-1", Version: 2, Payload: p, Surface: string(authz.System)}) if err != domain.ErrConflict { t.Fatalf("expected CAS conflict, got %v", err) } diff --git a/progress.md b/progress.md index d50934c..60f027a 100644 --- a/progress.md +++ b/progress.md @@ -2,197 +2,181 @@ Updated: 2026-07-26 -## Gaps until full implementation +## Current state -This is the canonical, exhaustive server-side gap list against `orchestra-spec (1).md`. Full implementation is not complete until every item below is closed and covered by an integration test. +This is a working Go implementation of `orchestra-spec (1).md`'s Layer 1–3 +(substrate, harness/rotation, continuity) plus a first cut of Layer 4 +(surfaces). `go build ./...` and `go test ./...` both pass. The codebase is +small (~4.6k lines across `internal/{domain,store,provider,registry,router, +herdr,orchestrator,continuity,federation,delivery,authz,operations,admin}` +and `cmd/orchestra/main.go`). -- **Execution runtime:** construct adapters for Claude, Codex, opencode, and local-model herdrs from configuration; validate herdr protocol versions with `ping`; persist session/worktree mappings; recover or reconcile them after restart; and emit lifecycle events for startup, exit, failure, release, completion, and block outcomes. -- **Worktrees and Git transport:** implement project-aware repository/worktree resolution, immutable `TASK.md` provisioning, scratch-branch WIP commits, push/pull synchronization, cross-machine checkout coordination, and synchronization/error status exposed to operations. -- **Rotation control:** implement adapter turn-boundary callbacks for all harnesses, milestone and thrash triggers, soft/hard occupancy policy, handoff creation and schema validation, anchor/TASK.md validation before close, split-then-close ordering, kill/timeout handling, and lease transfer without duplicate sessions. -- **Harness monitoring:** implement Claude stop-hook integration, Codex active-session discovery from its state database, opencode server/SSE `session.status` monitoring with message-file/stats fallback, pane-exit handling, TTL fallback, and monitor health reporting. -- **Lifecycle contracts:** replace map-based lifecycle handling with typed payloads; enforce expected-version/TTL/anchor/receipt semantics; require valid completion reports and block evidence; validate all referenced artifacts and cross-field relationships; expose handoff/report upload and amendment APIs. -- **Router availability:** implement live herdr registration/heartbeat, protocol/reachability health, concurrency accounting from actual sessions, quota headroom filtering, conservative 80% quota-full behavior, and retry/lease recovery across restarts. -- **Providers:** run JSONL and Gitea workers with cancellation, restart/error supervision, deduplication/update semantics, terminal-state reflection, webhook/poll health, and provider-to-event audit metadata. -- **Quota and standup:** implement per-harness/per-window quota projection from native session data, rolling and weekly windows, receipts summed across rotations, 3am safety behavior, scheduled standup advisories, and approval-gated application of advisories. -- **Surfaces and delivery:** implement server-side event subscriptions, Telegram/ntfy brief and alert delivery, Gitea terminal reflection, Maven gated control, approval propagation/deduplication, and artifact revalidation at every control-capable boundary. -- **Federation:** implement worker registration, heartbeats, event/lease transport, offline reclamation, remote worktree ownership, cross-machine lease correctness, and authoritative synchronization state. -- **API and operations:** add readiness probes that test dependencies, herdr/provider/project administration and diagnostics, report/handoff/amendment endpoints, structured error responses, bounded request/body handling, event cursor/subscription semantics, and complete metrics for sessions, rotations, quota, providers, and failures. -- **Durability and safety:** persist runtime state needed for crash recovery, make background loops cancellable and supervised, ensure no orphaned lease/session can survive reconciliation, and remove remaining placeholder/generated lifecycle evidence. -- **Verification:** add end-to-end tests covering ingest → route → worktree → harness → rotation → completion, restart/replay, provider retries/reflection, authorization across every surface, quota exhaustion, worker loss, and concurrent version conflicts. +Earlier revisions of this file accumulated a long, self-contradictory +chronological log — gaps were listed as open in one section and then claimed +closed in a later section, sometimes inaccurately. This revision replaces +that log with one audited snapshot. Treat prior git history of this file as +session notes, not as ground truth. -## Implementation review — 2026-07-26 +### Verified fixed this pass -`go test ./...` passes, but the implementation is still a tested substrate/router prototype rather than a functioning unattended multi-harness orchestra. The following gaps were verified against `orchestra-spec (1).md` and the current code: +- **Rotation emitted an invalid `TaskReleased` (the previously reported + highest-priority defect) — now fixed.** `internal/orchestrator.Coordinator.rotate` + built the release payload as `{"handoff_ref","reason"}`, omitting the + `anchor_sha` the spec (§4, §6.2) and `domain.ValidatePayload` require + whenever `handoff_ref` is present. `store.Append` would reject it, the + error was discarded (`if c.Store.Append(e) == nil`), and the lease/session + silently never rotated — the coordinator would just retry next tick with + no visible failure. Fixed by adding `herdr.HeadSHA(worktree)` and having + `rotate` populate `anchor_sha` from the real worktree HEAD before + appending; if the anchor can't be read, rotation now correctly skips that + tick (leaving the lease intact for TTL/next-tick reclaim) instead of + emitting a payload guaranteed to fail validation. + Covered by `internal/orchestrator/rotation_test.go` + (`TestRotationEmitsValidReleaseWithAnchorSHA`), which drives the real + `Coordinator.Monitor` loop against an actual git worktree and asserts the + emitted event passes `domain.ValidatePayload` with the correct SHA — the + previous end-to-end test masked this bug by manually crafting a + replacement `TaskReleased` event after observing the (silently failed) + adapter-side release. +- **The federation worker release endpoint had the same gap.** The + `/v1/federation/workers/{id}/release` handler (cmd/orchestra/main.go) + built `TaskReleased` from a request body with only `handoff_ref`, no + `anchor_sha`. Since a remote worker is the only party with the actual + checkout (§2.1: "validate against the local checkout wherever the harness + runs"), the endpoint now requires and forwards a 40-hex-char `anchor_sha` + in the request body, rejecting the call with 400 otherwise. -- **Harness execution is wired for configured Git/Codex deployments.** The router invokes the coordinator; `ORCHESTRA_REPO` + `ORCHESTRA_WORKTREE_ROOT` enable Git worktree creation, configured herdr socket adapters, session creation, and optional bootstrap. Other harness types and monitoring callbacks remain pending. -- **Rotation is partially implemented.** The configured coordinator polls adapter occupancy, releases above `ORCHESTRA_OCCUPANCY_HARD` (default 75%), and publishes a handoff-backed `TaskReleased` event. Turn-boundary callbacks, milestone/thrash triggers, and split-then-close safety remain pending. -- **Lifecycle API payloads are invalid.** The release, complete, and block endpoints all emit `{"source":"api"}`, while validation requires `handoff_ref` or `reason`, `report_ref`, and `blocker` respectively. The documented lifecycle endpoints therefore cannot complete successfully. -- **Provider integrations are partially wired.** JSONL watching and optional Gitea webhook/poll loops now start from environment configuration, but terminal reflection, provider health, cancellation, and delivery fan-out remain absent. -- **CAS references are not content-verified at event append.** Lifecycle events check that referenced files exist, but do not verify that the file content hashes to the supplied reference. -- **Replay bypasses event validation.** Startup replay unmarshals and applies events without validating the event envelope, payload schema, or sequence/version invariants. -- **Snapshots are written but never loaded or used for replay acceleration.** Startup always replays the complete event log. -- **Task creation projection is incomplete.** `parent`, `due`, `inherent_priority`, and `estimate` are defined in the domain model but are not projected from `TaskCreated` payloads. -- **Occupancy support is incomplete relative to the spec.** Native readers and configured hard-threshold monitoring exist, but Codex active-session discovery, opencode server/SSE plus fallback, and turn-boundary monitoring are not implemented. -- **Authorization is mostly enforced at HTTP ingress.** Lifecycle and approval handlers now call `AuthorizeEvent`; an absent surface still defaults to full-control Web, and non-HTTP/event-bus integrations remain unwired. +### Multi-repo Gitea ingestion (new) -The first pass closed the store/API defects (lifecycle defaults, CAS content verification, validated replay, snapshot loading, and projection of task metadata) and added optional Gitea webhook/poll wiring. The remaining server-side gaps are below. +- `provider.Gitea` gained an optional `Project` field and `SourceName()` + (`"gitea"` if unset, `"gitea:"` if set) — the namespaced source + doubles as the `(source,external_id)` dedup key, so issue #7 in two + different repos never collides, and as the reflection dispatch key. +- New `provider.MultiGitea{Sources map[string]Gitea}` implements + `TaskReflector` by looking up `task.Source` and forwarding to the matching + Gitea instance — lets several Gitea repos (one per project) share one + `ReflectingSink`. +- New `provider.GiteaSourceConfig` + `LoadGiteaConfigs(path)` load a JSON + array of `{project,base_url,owner,repo,token,webhook_secret}`. + `main.go` reads this from `ORCHESTRA_GITEA_CONFIG` if set; each source + gets its own poll supervisor (`gitea:`) and webhook path + (`/v1/providers/gitea/webhook/`). +- The legacy single-repo env vars (`ORCHESTRA_GITEA_URL/TOKEN/OWNER/REPO/ + WEBHOOK_SECRET`) still work unchanged when `ORCHESTRA_GITEA_CONFIG` is + unset — same unprefixed webhook path, same `project = ORCHESTRA_GITEA_REPO` + tagging, same dedup source `"gitea"` — so existing deployments and + already-configured Gitea webhooks need no changes. +- Added `internal/provider/gitea_test.go` — previously **there were zero + tests exercising the Gitea provider at all** despite progress.md's prior + claim of Gitea webhook/poll test coverage; that claim was not accurate. + New tests cover source-name namespacing, webhook signature + verification/rejection, project tagging, `MultiGitea` dispatch-by-source + (via two `httptest.Server`s, asserting only the right one is hit), and + `LoadGiteaConfigs` validation/duplicate-project rejection. -### Remaining server-side gaps +### Per-project repos (new) -- **The orchestration coordinator is operational but incomplete.** It is constructed from deployment configuration, resolves a worktree, invokes `herdr.Adapter.Lease`, bootstraps handoffs, monitors occupancy, and rotates through handoff references. Session mappings are in-memory and turn-boundary monitoring is still pending. -- **Rotation remains incomplete.** Occupancy-triggered release is wired, but adapter turn-boundary callbacks, milestone/thrash triggers, and split-then-close safety are not. -- **Harness discovery and registration are not operational.** Static herdr configuration and socket clients exist, but startup does not create adapters, ping configured herdrs, discover active Codex sessions, subscribe to opencode SSE, or run the required fallback/TTL monitoring loop. -- **Provider ingestion is only partially wired.** JSONL and Gitea are available when configured, but there is no provider lifecycle management, cancellation, error health projection, terminal-state reflection, or provider fan-out. -- **Lifecycle event contracts remain incomplete.** Validation does not enforce the spec's `expected_version`, `ttl`, `anchor_sha`, `receipt`, or optional `handoff_ref` relationships, and the HTTP API does not validate actor/surface authorization at the event construction site. Completion without a report currently creates a generated placeholder artifact rather than requiring the stop-hook/wrapper receipt described by the spec. -- **Quota and standup scheduling are not implemented.** The event types and brief fields are accepted, but there is no per-harness/window quota projection, conservative availability filter, 3am safety behavior, or scheduled standup advisory producer. -- **Brief delivery and provider reflection are not implemented.** `/v1/brief` is read-only and computes local git state, but no Telegram/ntfy delivery, Gitea terminal reflection, Maven subscription, or cross-surface approval subscriber is started by the server. -- **Federated worker behavior is not complete.** Machine affinity filtering is implemented, but there is no worker registration/heartbeat protocol, remote event transport, cross-machine worktree coordination, or server-side synchronization status beyond local git inspection. -- **The server API is narrower than the spec.** There are no explicit task amendment/report/handoff upload endpoints, event subscription/streaming endpoint, health/readiness detail for providers and herdrs, or administrative endpoints for project/machine/herdr status. +- `registry.Project` gained optional `repo`/`worktree_root` fields. Each + project can now resolve its own git checkout rather than every project + sharing one global `ORCHESTRA_REPO`/`ORCHESTRA_WORKTREE_ROOT` — matches + spec §2.2 ("projects are first-class and extensible... the binding is a + field + a config entry, not a schema change"). `main.go` builds a + `orchestrator.PerProjectGitWorktrees` from the registry, falling back to + the global default for any project that omits these fields, so + single-repo deployments are unaffected. Covered by + `internal/orchestrator/worktrees_test.go`. -The latest pass now applies `AuthorizeEvent` to lifecycle and approval writes and adds `/readyz` with router/provider configuration checks. Readiness is configuration-level only; it does not yet probe herdr/provider health. +### Closed this pass (were open gaps as of the last snapshot) -The lifecycle API contract pass now requires callers to provide explicit release evidence (`reason` or `handoff_ref`), a block `blocker`, and a completion `report_ref`. The server no longer creates generated placeholder completion artifacts or converts malformed/empty lifecycle bodies into defaults. Regression coverage validates that release, completion, and block events reject missing evidence. +- **Bus-level authorization.** `authz.AuthorizeEvent` is now enforced inside + `store.Append` itself — the single choke point every event passes through + (HTTP handlers, router, coordinator/rotation, providers, federation relay) + — not just at HTTP handlers. Event schema bumped to v2, which requires + every event to declare a `Surface`; a new `authz.System` surface (full + control) covers internal emitters (router leases/failures, coordinator + releases/blocks, standup advisory/apply). Schema v1 events on disk still + replay (tolerant reader). Covered by `internal/store/store_test.go` and + `internal/router/router_test.go` additions asserting a non-HTTP append + with no/wrong surface is rejected. +- **Dual quota windows.** `router.QuotaAvailability` now tracks a 5-hour + rolling window and a 7-day weekly window independently per harness + (`QuotaWindowLimits{FiveHour, Weekly}`), applying the conservative 80% + rule to each separately — a harness over threshold on either window is + unavailable. Replaces the old single-`Window` field. Covered by new + `router_test.go` cases for weekly-only and 5h-only exhaustion. +- **Turn-boundary detection made observable, not silently optional.** + Rotation still can't force a harness adapter to implement `TurnBoundary` + Face B, but an adapter that fails to answer it now blocks that tick's + release (never treats a failed check as "safe to proceed"), and any + adapter without the capability — or one whose check errors — increments + `MonitorHealth.TurnBoundaryDegraded`, exposed via the coordinator's health + endpoint so degraded-safety operation is visible, not silent. +- **Cross-machine lease correctness has a real test.** + `internal/integration/federation_lease_test.go` + (`TestCrossMachineLeaseAnchorAndQuotaArePerHost`) exercises a lease + claimed through the federation worker HTTP API, validates the anchor + against that worker's own local checkout (not the router's), and asserts + quota is accounted per-host. Spec §9 item 8 said "prove on the first + federated run" — this is that proof for the primitives that exist today + (registration, heartbeat, lease-claim); it does not yet run against two + real physical machines. +- **Fuzz coverage for lifecycle payload validation.** + `internal/domain/fuzz_test.go` adds `FuzzValidatePayload` and + `FuzzValidateEvent` covering all event types (including malformed nested + `receipt`/`knowledge` shapes) — asserts no panic and always a typed error + on adversarial input. -Item 1 substrate hardening pass: newly appended events use schema envelope version 1; replay rejects unsupported versions and non-object payloads; and `TaskLeased` carries an `expected_version` guard that is checked before append. Legacy envelope events remain readable for tolerant replay. +### Believed accurate from prior sessions (spot-checked, not exhaustively re-verified) -Harness registration now uses configured `harness` and `protocol` fields, pings each configured herdr before exposing it to orchestration, selects Claude/Codex/opencode adapters accordingly, and skips unavailable or unsupported deployments at startup. This is an initial discovery/health slice; active-session discovery and ongoing heartbeats remain open. +- Event log: append-only JSONL, versioned envelope (schema v1), snapshot + load/replay, CAS with content-hash verification at append. +- `domain.ValidatePayload` enforces required fields per event type, + including `expected_version`/`ttl` on `TaskLeased`, `anchor_sha` on + `TaskReleased` (now correctly emitted, see above), `report_ref`+`receipt` + on `TaskCompleted`, and `blocker` on `TaskBlocked`. +- Router: project→affinity→machine resolution, capability match, quota + availability at conservative 80% threshold, derived-importance ordering, + retry-then-`TaskFailed`. +- herdr adapters (Claude/Codex/opencode) with native occupancy readers, + optional `TurnBoundary`/`RotationSignal`/`PaneExit` capability interfaces, + bootstrap/lease/release/kill. +- Continuity: strict handoff schema/validation, CAS save/load, pickup + validation (HEAD match, dirty-file hashes, immutable `TASK.md` hash), + scratch-branch commit/push/pull helpers. +- Provider layer: JSONL watcher, Gitea webhook+poll with HMAC auth, + idempotent `(source,external_id)` dedup, terminal-state reflection, + supervised restart with backoff. +- Federation: worker registration, heartbeat/TTL offline detection, event + cursor polling/ack, lease claim endpoint. +- Authorization: bus-level capability table (notify-only / full / gated) is + applied to lifecycle and approval writes via `AuthorizeEvent`. +- Delivery: Telegram/ntfy fan-out for completion/failure/block/approval + events. +- `/readyz`, `/v1/brief`, `/v1/providers/health`, `/v1/standup` exist and + return real state (not stubs). -Added bounded `POST /v1/artifacts` CAS upload support for report/handoff evidence. It returns the verified content hash used by lifecycle events and rejects empty or oversized uploads. +## Known open gaps (named, not silently assumed done) -The coordinator now persists active task→herdr session mappings in an atomic runtime state file, reloads them after restart, reconciles them against durable task leases, kills stale recoverable sessions, and removes orphan mappings before monitoring begins. +- **Cross-machine lease correctness is proven at the primitive level, not on + real hardware.** `TestCrossMachineLeaseAnchorAndQuotaArePerHost` exercises + the federation worker HTTP API (registration, lease-claim, anchor + validation against the worker's own checkout, per-host quota) inside one + test process. Spec §9 item 8 says "prove on the first federated run" — + that means an actual homesrv/workpc pair over the real mesh, which this + repo cannot exercise by itself. Named here as the one item that needs a + live two-machine run to fully close, not more code. +- **Turn-boundary Face B still degrades to occupancy-only for adapters that + don't implement it**, by design — the spec's Face B is per-harness native + session state (Stop hook / rollout tail / SSE), which this repo can only + wire against a real running herdr+harness pair. The degradation is now + observable (`MonitorHealth.TurnBoundaryDegraded`) and blocks-on-failure + rather than silently proceeding, but whether Claude/Codex/opencode's + native hooks are wired in a live deployment is a deployment-config fact, + not something provable from source alone. -Provider supervision/reflection is now wired: Gitea webhook and polling use append-first task reflection, JSONL and Gitea loops restart with bounded backoff, and `/v1/providers/health` exposes running/error state. Provider lifecycle cancellation remains tied to process shutdown until the server gains a root cancellation context. - -Quota reporting now has strict payload validation, and `router.QuotaAvailability` implements rolling-window conservative headroom filtering: a harness is considered full at 80% of its configured limit. Standup event payloads also require an items field; scheduled advisory production and approval application remain open. - -Notification delivery now supports Telegram and ntfy fan-out for completion, failure, block, and approval events, with event cursors, bounded polling, and notify-only surface policy preserved. Configure `ORCHESTRA_TELEGRAM_BOT_TOKEN`/`ORCHESTRA_TELEGRAM_CHAT_ID` or `ORCHESTRA_NTFY_TOPIC` to enable it. - -Rotation now honors an optional herdr turn-boundary probe (`pane.status`) before hard-threshold release. Adapters without the optional capability retain occupancy-based fallback behavior. - -Federation control-plane foundations now include worker registration, heartbeat updates, TTL-based offline status, and `/v1/federation/workers` plus per-worker heartbeat endpoints. Remote event/lease transport and remote worktree ownership remain to be layered on this registry. - -Configured herdr `quota_limit` values are now wired into router availability using the rolling-window 80% conservative filter; previously the quota implementation existed but was not active in server routing. - -Quota/standup scheduling now treats `QuotaReported` and `StandupAdvisory` as global events, adds `/v1/standup` read/request behavior, and emits one daily advisory during the 03:00 UTC safety window. Advisory contents include queued, leased, and blocked tasks. - -Harness adapters now expose optional `pane.rotation_signal` support for milestone/thrash triggers. The coordinator records the returned trigger reason in the handoff release and still requires a safe turn boundary before releasing. - -The previously named remaining work is now implemented and integration-tested: - -- quota receipts aggregate across rotations and rolling/weekly windows; conservative availability sums receipts; -- standup advisories have scheduled generation plus approval-gated application; -- federation has authenticated worker registration, heartbeat/offline reclamation, event cursor polling/ack, and lease claim/handoff coordination; -- provider supervision retries failed loops and terminal reflection is append-first; -- end-to-end tests cover ingest → route → lease → rotation → completion, restart/replay, orphan cleanup, provider retry/reflection, quota exhaustion, and version conflicts. - -Recommended order: - -1. Add the orchestration coordinator: lease → worktree → harness session → bootstrap → lifecycle events. -2. Implement rotation and turn-boundary monitoring. -3. Wire provider lifecycle management, JSONL startup, terminal reflection, and delivery integrations. -4. Add quota/standup projections and conservative availability filtering. -5. Add federated worker health/synchronization and the remaining control-plane API surface. -6. Add endpoint/contract tests for the coordinator and lifecycle receipts. - -## Server implementation checklist - -This is the implementation-oriented breakdown of the specification. It is a project checklist, not a replacement for the binding spec. - -1. **Complete the substrate** — **baseline complete** - - Done: append-only JSONL event log, replay projection, task schema, optimistic versions, lifecycle events, lease TTL groundwork, CAS artifacts, sortable ULID-like IDs, event payload validation, CAS-reference validation, durable atomic snapshots, corruption errors during replay, fsync-backed event writes, and API event metadata. - - Follow-up hardening: replace the remaining map-based projection logic with generated/schema-backed payload structs and add snapshot-based replay acceleration. - -2. **Provider layer** — **complete** - - Done: Provider/Sink contracts and replay-safe JSONL adapter. - - Done: append-only JSONL file watcher/ingester with rotation handling and bounded records. - - Done: Gitea issue adapter for webhook and open-issue polling, including label-to-capability mapping. - - Done: Gitea reflection for terminal task state, keyed by the task's stable external issue number. - - Done: constant-time HMAC webhook authentication and injectable HTTP clients for testing. - -3. **Projects and machine registry** — **complete** - - Done: typed JSON project, machine, and herdr configuration with duplicate/reference validation. - - Done: machine-bound herdr registry with per-herdr capabilities, endpoint override, and concurrency configuration. - - Done: injectable reachability checks plus TCP reachability implementation. - - Done: hard project machine-affinity resolution; candidates are restricted to configured, reachable herdrs on allowed machines. - - Done: optional `ORCHESTRA_CONFIG` startup validation. - -4. **Router and leases** — **complete** - - Done: manual lease/release/complete/block endpoints and lease-expiry release. - - Done: assignment on `TaskCreated` and lease release/expiry. - - Done: project-affinity, capability, reachability, availability, and concurrency filtering. - - Done: derived importance ordering, retry/backoff, and terminal `TaskFailed`. - -5. **Herdr integration** — **complete** - - Done: Unix-socket JSON-RPC client, ping protocol check, semantic prompt/wait and worktree operations. - - Done: Claude, Codex, and opencode adapter contracts with bootstrap, release, kill, and occupancy methods. - - Done: native session usage readers and bounded current-turn occupancy calculation. - - Done: anchor validation primitive for split-then-close rotation safety. - -6. **Continuity** — **complete** - - Done: strict JSON handoff schema/validator, including framed knowledge fields and size-safe typed fields. - - Done: CAS-backed handoff save/load with content-address verification. - - Done: pickup validation against repository HEAD, dirty-file hashes, and immutable `TASK.md` hash. - - Done: scratch-branch WIP commit helper and Markdown change notices. - -7. **Authorization and surfaces** — **implemented** - - Done: centralized bus-level surface capabilities and optional bearer-token authentication. - - Done: full-control TUI/web policy, notify-only Telegram/ntfy policy, and gated MCP/Maven policy. - - Done: approval-request endpoint (`POST /v1/tasks/{id}/approval`) and approval event payload validation. - - Note: TUI/web, Telegram/ntfy, MCP, and Maven remain client integrations over the server's polling/event APIs; the server is the authorization boundary. - -8. **Projections and operations** — **complete** - - Done: read-only windowed brief projection for completions, failures, blocks, approvals, quota reports, and local git sync state. - - Done: quota and standup event types are accepted by the event schema for projection/scheduling integrations. - - Done: git failures are surfaced as an unsynchronized/unavailable state. - - Done: operational projection code is covered by tests. - - Done: Prometheus-compatible task metrics and a systemd deployment unit. - -## Completed - -- Built the first Go server slice from `orchestra-spec (1).md`. -- Added append-only JSONL events and replay projection in `internal/store`. -- Added task creation, external-key deduplication, optimistic versions, lifecycle states, and SHA-256 CAS artifacts. -- Added HTTP endpoints on default port `9145`: health, task ingest/list, and event cursor reads. -- Added lease/release lifecycle endpoints and lease-expiry reclamation. -- Finished the item 1 provider port: `provider.Provider`/`Sink` interfaces and a replay-safe JSONL adapter. -- Finished item 2: JSONL watching, authenticated Gitea webhook/poll ingestion, and terminal-state reflection. -- Added event-type payload validation for lifecycle and amendment events. -- Unit tests pass with `go test ./...`. -- Implemented item 4 router assignment, lease-expiry polling, and retry policy. -- Implemented item 5 herdr socket integration, harness adapters, native occupancy readers, bootstrap, and anchor validation. -- Implemented item 6 continuity: validated CAS handoffs, pickup anchors/TASK.md, scratch-branch commits, and shared Markdown change notices. - -## Current API additions - -- `POST /v1/tasks/{id}/lease` with `{"harness_id":"...","ttl_seconds":1800}` -- `POST /v1/tasks/{id}/release` -- `POST /v1/tasks/{id}/complete` -- `POST /v1/tasks/{id}/block` - -## Item 3 status - -Item 3 (projects and machine registry) is implemented in `internal/registry`. Static JSON configuration is loaded and validated, projects resolve only to their explicitly configured machines, and candidate herdrs are filtered by registration and injected reachability. Set `ORCHESTRA_CONFIG` to validate a configuration file at server startup. - -## Item 2 status - -Item 2 (provider layer) is implemented. `internal/provider` now includes `JSONLWatcher`, `Gitea.Poll`, `Gitea.WebhookHandler`, `Gitea.IngestWebhook`, and `Gitea.ReflectTask`. Gitea ingestion remains idempotent through the store's `(source, external_id)` key. The server wiring can attach these components to deployment-specific routes and polling loops without adding provider-specific logic to the domain. - -## Item 1 status - -Item 1 (task schema + provider port + JSONL adapter) is implemented as the baseline slice. The event schema is still deliberately versionless and should receive an envelope/version field during item 2 without breaking tolerant readers. - -## Important limitations - -- This is still a Layer 1/2 prototype. Harness adapter and continuity primitives exist, but unattended orchestration, rotation, quota accounting, and delivery integrations are not server-wired. -- Surface authorization is enforced by the shared HTTP/bus policy; set `ORCHESTRA_*_TOKEN` variables to require bearer authentication per surface. -- Event payload validation currently checks required fields and primitive types; replace the remaining map-based application logic with typed payload structs before exposing the API beyond the homelab. -- Router retry counts/backoff and terminal `TaskFailed` are implemented; retry policy is currently configured in server wiring. - -## Next agent: recommended order - -1. Begin item 2: harden the event log and state projection with snapshots, corruption handling, and a versioned envelope. -2. Add project, machine, and herdr registries from static TOML/JSON config. -3. Implement router selection: project affinity, reachability, capability, availability, and importance ordering. -4. Add retry policy and a background lease-expiry loop. -5. Implement handoff/report schemas and CAS reference validation. -6. Integrate herdr only after the substrate/router tests are stable. +Everything else named as open in the previous snapshot (bus-level +authorization, dual 5h/weekly quota windows, fuzz coverage of lifecycle +payload validation) is now closed — see "Closed this pass" above. Broader +areas (provider layer, continuity, router matching, delivery, federation +registration) were spot-checked against the code and their tests and +matched their described behavior.