diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..257f268 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,4 @@ +.git +data +web/node_modules +web/dist diff --git a/.gitignore b/.gitignore index 0a11ee3..d2a3e01 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,19 @@ -# fill later +# Private, composed deployment configuration. Install into /etc/orchestra/ +# only after all environment-specific values have been filled in. +.orchestra-config/ + +# Built binaries (never commit — a stale committed binary is a deployment- +# confusion hazard). +/orchestra +/orchestra-worker + +# Web UI build inputs/outputs. node_modules in particular ships vendored Go +# packages (e.g. flatted/golang), so leaving it merely untracked is not +# enough — `go build ./...` and `go test ./...` walk into it. `web/go.mod` is +# the fix: it ends the parent module's package graph at that directory. +/package-lock.json +node_modules/ +.node_modules/ +web/dist/ +web/tsconfig.tsbuildinfo +build/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..7aefa28 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,147 @@ +# Orchestra + +A Go implementation of `orchestra-spec (1).md` — an unattended multi-agent +task orchestrator that leases coding tasks to CLI harnesses (Claude Code, +Codex, opencode) running inside `herdr`-managed panes, rotates them across +context-window limits, and hands off work via a git-anchored continuity +protocol. + +Layout: `internal/{domain,store,provider,registry,router,herdr,orchestrator, +continuity,federation,delivery,authz,operations,admin}` + `cmd/orchestra/main.go`. + +## Ground truth over documentation + +This repo has a documented history of code that *looks* wired but isn't — +packages with tests that pass in isolation while the live call path silently +no-ops (bare `continue` on error, discarded return values). See `AUDIT.md` +for the full audit; it is also the running log (there is no separate log +file). **Before trusting a claim in `AUDIT.md` that something "works" +or "is fixed," check the actual call site** — the file is written by past +sessions of this same assistant and has previously overstated completion. + +The single most reliable way to verify herdr-adapter code is right: don't +read `internal/herdr/adapter.go` and assume the method names are real. Ping +the live herdr instance and check. + +## herdr protocol — verified against a live instance, 2026-07-27 + +- herdr speaks JSON-RPC over a raw TCP (or unix-socket) connection — **not + HTTP**. `internal/herdr/herdr.go`'s `Client.Call` is the only correct way + to talk to it; a bare `curl` to the port returns nothing. +- Request shape: `{"id":"","method":"","params":}`. Herdr's + Rust JSON-RPC decoder requires `params` to be present and rejects a bare + `null` — always send `{}` for parameterless calls (the client does this + automatically). +- Full real method list is committed at `deploy/herdr-schema.json`, captured + live from `192.168.1.105:9245` (the `workpc` herdr) since no local `herdr` + CLI is available in this sandbox — the schema was reconstructed by sending + an unknown method name and reading the `unknown variant ... expected one + of ...` error, then probing each method of interest with `params:{}` / + `params:{pane_id:"nonexistent"}` to read Rust serde's `missing field + ` errors for its param shape. +- **Confirmed invented (do not use, they don't exist):** `pane.release`, + `pane.kill`, `pane.rotation_signal`, `pane.status`. If you see these + anywhere, it's a bug, not a valid call. +- **Real replacements:** `pane.close({pane_id})` for kill; + `pane.release_agent({pane_id, source, agent})` for release (structurally + different — does *not* return a `handoff_ref`, see below). No replacement + exists for `rotation_signal` — herdr has no concept of Orchestra rotation. +- **Architectural point that's easy to get wrong:** herdr never produces a + handoff. The agent writes the handoff artifact (§6.1 of the spec); herdr's + role in "release" is only to drop its own claim on the pane/agent binding. + Any adapter code that expects herdr to hand back a `handoff_ref` is wrong + by construction, independent of whether the method name is right. +- Protocol version is returned as a **JSON number** (`17`), not a string, + even though `config.jsonc` declares `"protocol": "17"` as a string. + `CheckProtocol`'s raw-bytes fallback happens to make this compare correctly + today — don't "clean up" that code without checking this note first, or it + might start doing a real numeric-vs-string comparison and break. + +## Deployment topology (as of 2026-07-31) + +- **Runs under Docker Compose, not systemd.** `docker compose -f compose.yaml + -f compose.live.yaml` in `/home/kami/docker-apps/orchestra-web-ui`, building + both images from this repo: `orchestra-api` (bound `0.0.0.0:9145`, which is + intentional — ufw restricts the port to one other LAN machine) and + `orchestra-web-ui` (nginx proxy, `127.0.0.1:19145`). Logs are + `docker logs orchestra-api`. **Deploying a code change means rebuilding the + compose images** (`up -d --build`) — the running image can silently predate + recent commits, so compare its build time against `git log`. +- `orchestra.service` was the previous deployment; its unit file and + `redeploy.sh` were deleted from `deploy/` on 2026-07-31. A stale installed + copy must stay stopped — it binds the same port and data dir as the + container. `orchestra-worker.service` is a *different*, still-current unit. +- Config: env vars come from **`.env` in the compose directory** via + `env_file:`; only `config.jsonc` is bind-mounted into `/etc/orchestra/`. + There is no container entrypoint script — `Dockerfile.api` execs + `/app/orchestra` directly. Neither deployed file is the repo's + `deploy/config.example.jsonc`. +- Browser operator accounts live in `$ORCHESTRA_DATA/auth.db`. Create or reset + one with `orchestra-user set -data /data -username NAME` while the API is + stopped, or use the authenticated Settings screen. The old + `ORCHESTRA_WEB_USERNAME`/`ORCHESTRA_WEB_PASSWORD_HASH` pair is accepted only + for a one-time import into an empty database and should then be removed. +- Two machines in the registry: `homesrv` (192.168.1.104) and `workpc` + (192.168.1.105), each nominally running 3 herdrs (claude/codex/opencode). + In practice **homesrv has no local herdr running** (connection refused on + 9245) — only workpc's herdr is live and reachable. `main.go` only logs + herdr connection *failures* at startup, never successes, so "no log line" + for a herdr does not mean it's down — check reachability directly. +- There was a real, live, stuck task as of 2026-07-27: workspace `wA`, task + id `06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode harness, pane `wA:p1`, + `agent_status: "blocked"`. Likely stuck because rotation/release could + never reach it (B2/B5). Check whether it's still stuck before assuming + fixes here have taken effect operationally — code fixes don't retroactively + unstick an already-orphaned pane; that needs a manual kill/restart once the + release path is trustworthy. + +## Federation — Design B is the live design (as of 2026-07-30) + +**Design B** ("workers pull tasks", `/v1/federation/*`) is the live design and +has a real client: `cmd/orchestra-worker/main.go` (~1,131 lines, with tests in +`cmd/orchestra-worker/main_test.go`) is the deployed worker — the workpc +OpenCode worker runs it. Build new cross-machine work on Design B. + +The **Design A guardrail has landed**: `Coordinator.adapterFor` +(`internal/orchestrator/orchestrator.go`, the `LocalHerdr` check) refuses to +resolve an adapter for a session owned by a non-local herdr, returning +`session %s is owned by non-local herdr %s` instead of validating a git anchor +(`git rev-parse HEAD`) against the wrong machine's checkout. Rotation/cleanup +therefore no longer act on remote leases. + +**Design A is gone (deleted 2026-07-31).** `clients/herdr-bridge.go` ("drive +the remote socket": homesrv calling `worktree.create`/`agent.start` directly on +workpc's herdr over TCP as if it were local) was deleted along with the whole +`clients/` directory — the operator confirmed it is undeployed now that workers +carry cross-machine work, which superseded the 2026-07-27 "keep through Phase +5" decision. There is no bridge to preserve; do not reintroduce +coordinator-side calls to a remote herdr socket. + +**Completion is worker-owned.** `orchestra-worker` watches for an +`.orchestra/done` marker, confirms via `AgentStatus` that the agent is no +longer busy, then posts through `/v1/federation/*` with the lease epoch and +expected version. The old harness-hook path — `.orchestra-report.md` plus +`POST /v1/harness/complete` — is **deleted**, endpoint, handler, and +`deploy/hooks/` scripts alike. `/v1/harness/turn` remains for turn-boundary +decisions. + +## Working conventions + +- **workpc worker deployment target:** copy the built worker binary to + `workpc:~/orchestra-deploy/orchestra-worker` (that is, + `/home/kami/orchestra-deploy/orchestra-worker`), not directly to + `/usr/local/bin`. The workpc deployment process installs from this staging + path. Verify the remote checksum and Go build revision before restart. +- `go build ./...`, `go vet ./...`, and `go test ./...` must all pass — `go + vet` was broken for a while (duplicate JSON struct tags) and nobody + noticed because only `build`/`test` were being checked. Always run all + three. +- Silent `continue`-on-error is the recurring bug pattern in this codebase + (adapter lookups, rotation, expiry). When touching `internal/orchestrator` + or `internal/herdr`, prefer a recorded/observable failure + (`MonitorHealth` fields) over a bare `continue` — that's literally what + turned B1/B2 invisible for as long as they were. +- Don't invoke destructive herdr calls (`pane.close`, `pane.release_agent`) + against a real pane from an investigative/audit session without asking + first — there is live operator state on the other end (see the stuck-task + note above). diff --git a/AUDIT.md b/AUDIT.md new file mode 100644 index 0000000..acfa298 --- /dev/null +++ b/AUDIT.md @@ -0,0 +1,427 @@ +# Orchestra audit — handoff first + +## 2026-08-11 — tmux backend and Claude in-place context rollover + +- `orchestra-worker` now selects `ORCHESTRA_WORKER_BACKEND=herdr|tmux`. + The tmux backend is intentionally Claude-only: it creates one detached, + isolated tmux session per task, resolves configured tmux pane indexes rather + than assuming `0.0`, starts Claude in the task worktree, handles the exact + workspace-trust prompt, captures the pane, forwards explicit input, reports + health/status, and retains/kills the session at the existing lifecycle + boundaries. Codex and OpenCode still use herdr. +- Registry entries accept `"backend":"tmux"`. Those entries are worker-owned + even on the coordinator machine, so the Dockerized coordinator neither + probes nor tries to operate a host tmux socket. Their router health remains + gated by a fresh worker heartbeat and local backend check. +- Claude rotation no longer asks the agent to duplicate the workpc hook's + handoff in `.orchestra-handoff-report.md`. A changed `HANDOFF.md`, observed + after Claude is idle, drives the persisted in-place sequence `/clear` + + `ENTER`, then `@HANDOFF.md` + `ENTER`; the exhausted transcript identity is + discarded so the next observation resolves the new session. The canonical + cross-worker release protocol remains separate and unchanged. +- The portable workpc Claude configuration (`settings*.json`, `CLAUDE.md`, + status line, hooks, and helper scripts) was checksum-verified after copying + to homesrv. The prior homesrv files are backed up under + `/home/kami/.claude/backups/workpc-import-20260811T120000`; machine/session + state was excluded and local-only hooks were preserved. +- Verified in the source tree with the real isolated-tmux lifecycle test, + focused backend/worker tests, `go build ./...`, `go vet ./...`, + `go test ./...`, `go test -race ./...`, and the web lint/test/production + build. No worker binary, registry file, service, container, or live pane was + deployed/restarted/touched; live Claude/tmux behavior remains a deployment + QA item rather than a completed claim. + +Audited 2026-07-30 against the working tree, spec, deployed coordinator, +workpc worker, event log, and live herdr (read-only). + +**Verdict:** the source-level P0/P1/P2 fixes are in place, the live +coordinator can replay its repaired event log, and the current OpenCode worker +is connected. The system is not safe to run unattended until the controlled +QA matrix has passed for all three harnesses. + +## Evidence + +- `go build ./...`, `go vet ./...`, `go test ./...`, and `go test -race ./...`: pass. +- Live B17: release `seq=251`, re-lease `252`, completion `262`; the simple + probe needed six approvals, logged a `409 lease version conflict`, and + recorded `consumed:0`. +- Live follow-up: Docker owns the coordinator; the old systemd unit is + inactive. The pre-v2 repeated-`seq=1` event prefix was migrated with a + backup-preserving, explicit tool before the current coordinator replayed it. +- Deployment follow-up: coordinator and installed workpc OpenCode worker are + clean revision `d6cab133b56666f81f569f4c1c3c9a6f104088d1`. The worker service + restarted at 2026-07-30 16:07 +04, has emitted no federation failures since, + and its configured Unix-socket herdr answered `ping` with protocol `17`. + +## Federation design status (updated 2026-07-30) + +Design B ("workers pull tasks", `/v1/federation/*`) is the **live** design. It +is no longer clientless: `cmd/orchestra-worker/main.go` (~1,131 lines, tests in +`cmd/orchestra-worker/main_test.go`) is the deployed worker, and the workpc +OpenCode worker runs it. Any earlier statement here or in `CLAUDE.md` that +Design B "has zero clients — no worker binary exists" is obsolete. + +The Design A guardrail from the 2026-07-27 decision has landed: +`Coordinator.adapterFor` (`internal/orchestrator/orchestrator.go`) refuses to +resolve an adapter for a session owned by a non-local herdr, so rotation and +cleanup can no longer validate a git anchor against the wrong machine's +checkout. + +Design A is **deleted as of 2026-07-31**, superseding the 2026-07-27 "retain +through Phase 5" decision: the operator confirmed the bridge is undeployed now +that workers carry cross-machine work, so `clients/` was removed rather than +tracked. The Phase 6 cutover is therefore already done on this axis. + +The legacy harness-hook completion path was removed in the same pass, once it +was confirmed that nothing calls it. `orchestra-worker` owns completion — it +watches for `.orchestra/done`, confirms via `AgentStatus` that the agent is not +busy, then posts through `/v1/federation/*` with the lease epoch and expected +version. Deleted: the `/v1/harness/complete` route (a `410` stub), its unmounted +`harnessCompletion` handler, that handler's test (green against unreachable +code — the pattern this audit exists to catch), and the three `deploy/hooks/` +scripts, which still used the older `.orchestra-report.md` marker and would +have failed against the `410`. `/v1/harness/turn` is unaffected and still live. + +Note for the QA matrix: the OpenCode run completed through the worker path, so +no hook script was exercised. Nothing about hook-based completion was ever +verified live, which is why deleting it costs nothing. + +## Remaining release blockers + +- **Only OpenCode capacity is ready.** Workpc's `workpc-opencode` worker has + a configured project file and a reachable local herdr Unix socket. Homesrv + has no reachable herdr, and no Claude/Codex worker/herdr pair has been + verified, so the full three-harness matrix cannot begin yet. +- **B17 needs a fresh controlled run.** The historical probe's six approvals, + one `409 lease version conflict`, and `consumed:0` receipt came from the + old worker. They cannot be treated as evidence for the current worker until + a live OpenCode run is repeated; Claude and Codex require their own runs. + +## QA handoff — next agent + +1. **Preflight before creating work.** Read `GET /v1/federation/workers` and + coordinator diagnostics through an authenticated operator session. Confirm + each target worker reports revision `d6cab13`, supported `test-e2e`, and + fresh `herdr_status: reachable`; raw-ping its configured local Unix socket + with `params:{}` and confirm protocol 17. Confirm no pre-existing agents + or leased task on the target harness. +2. **OpenCode controlled continuity run.** Submit one new disposable + `test-e2e` task that makes a deterministic marker, releases at a clear turn + boundary, validates pickup from the resulting anchor, then completes. + Record event sequence, handoff ref, anchor SHA, transaction id, lease epoch, + native session evidence, quality-gate result, remote SHA, and a receipt + with known non-zero (or explicitly explained known-zero) usage. Do not use + destructive herdr calls against unrelated panes. +3. **Exercise rotation and recovery.** In separate disposable tasks, trigger + soft, hard, milestone, thrash, coordinator restart, worker restart/lost + response, stale completion, and corrupted-worker-state paths. Verify each + result is a fenced lifecycle event or durable `needs_attention`, never a + silent retry or orphaned pane. Preserve the predecessor until matching + pickup validation. +4. **Repeat on Claude and Codex only after provisioning their own reachable + worker/herdr pairs.** Do not treat OpenCode evidence as cross-harness + proof. After all runs, compare worker/coordinator revisions and checksums, + attach the artifacts/event ranges to this audit, and only then clear the + live release gate. + +## P0 — correctness + +| ID | Current failure | Required fix | +|---|---|---| +| H1 | **Closed 2026-07-30.** The checkout-owning worker and coordinator turn path now use `RotationStateMachine`. Workers persist harness-native identity (Claude/Codex transcript, OpenCode SQLite session id), apply soft/milestone/thrash/hard-boundary decisions, and record unknown activity/occupancy/boundary as degraded health rather than zero usage. | Verified by `go test -race ./...`; the existing turn-policy coverage now exercises the shared state machine. | +| H2 | **Closed 2026-07-30.** `PrepareRelease` verifies immutable `TASK.md`, checkpoints all repository work except protocol markers, always pushes the per-task project's scratch anchor, verifies it with `ls-remote`, and only then seals the CAS handoff. | `TestScratchCommitCapturesAllGitStatesExceptProtocolMarkers` covers staged, deleted, renamed, untracked, and protocol-marker cases; release uses the configured project remote. | +| H3 | **Closed 2026-07-30.** Worker state persists idempotent release transactions through `prepared → anchor_pushed → event_committed → pickup_validated → predecessor_retired`. Release/pickup endpoints bind transaction, anchor, and lease version; a predecessor remains mapped and is retired only after matching pickup validation. | `TestReleaseTransactionSurvivesReLeaseUntilMatchingPickup` covers transaction propagation and pickup epoch binding; full race suite passes. | +| H4 | **Closed 2026-07-30.** Every new lease carries an opaque durable `lease_epoch`; renew/release/pickup/complete validate the exact harness owner and epoch at the store boundary and federation API. Offline heartbeats retain leases until expiry, new workers require a fresh reachable local-herdr probe, and local/worker ownership loss stops or durably quarantines the old pane before its mapping is dropped. | `TestLeaseEpochFencesStaleOwnerLifecycleWrites`, `TestAvailableRequiresFreshReachableLocalHerdrHealth`, plus the full race suite cover stale re-lease/completion and health admission. | +| H5 | **Closed 2026-07-30.** `Store.Append` validates legal state/owner/epoch transitions, fsyncs the event before applying its projection, and replays projections solely from `events.jsonl` (snapshots are disposable caches). CAS, worker/federation/coordinator state use temp-file + fsync + rename; corrupt worker state aborts startup. The legacy `/v1/harness/complete` route and its handler were deleted outright on 2026-07-31 (previously a 410 stub plus an unmounted, separately-fenced handler). | `TestOpenRebuildsOnlyFromLogAndIgnoresCorruptSnapshot`, `TestWorkerRefusesCorruptDurableState`, and `go test -race ./...` pass. | + +## P1 — autonomy and recovery + +- **Recovery:** **Closed 2026-07-30.** Launch/recovery faults now emit + `TaskNeedsAttention`, retaining the durable harness owner and lease epoch. + Renew, release, expiry, and a late reconciled completion accept that same + fenced lease; worker state advances its expected aggregate version without + dropping the live session. `TaskBlocked` remains terminal for an explicit + operator block. `TestNeedsAttentionRetainsFencedLeaseForLateCompletion` + covers the durable recovery path. +- **Retries:** **Closed 2026-07-30.** Hand-off-less `TaskReleased` is the + single durable reclaim transition. It projects exponential `attempt`, + `next_retry_at`, and `failure_class`; router assignment reads those fields, + so coordinator restarts cannot reset a backoff or retry limit. + `TestReclaimPersistsAttemptAndBackoffAcrossReopen` covers replay. +- **Launch:** **Closed 2026-07-30.** Workers emit a fenced + `TaskLaunchAcknowledged` only after a local start/prompt is persisted. + Typed NACKs immediately reclaim transient unusable capacity, terminally + block invalid handoffs, and retain uncertain live panes for reconciliation. +- **Completion:** **Closed 2026-07-30.** `.orchestra/done` is explicit + intent only; the worker also requires native non-busy identity, runs its + quality gate, verifies immutable `TASK.md`, commits, pushes, and checks + the remote SHA before it emits completion. +- **Quota:** **Closed 2026-07-30.** Completion receipts contain native + per-lease deltas plus a known/unknown marker. Five-hour and weekly + projections are published from the same receipts; any bounded harness + without fresh known usage fails routing closed. +- **Approvals:** **Closed 2026-07-30.** Projects have a validated audited + `safe_operations` policy limited to worktree-local read/edit/test/Git. + Workers inject it into the task prompt; network, secrets, destructive + actions, and paths outside the worktree remain operator-gated. +- **Observability:** **Closed 2026-07-30.** Task projections now retain + lifecycle phase, last error, retry time/failure class, lease epoch, pane + state, and anchor. Release/anchor certification faults enter durable + `needs_attention` instead of disappearing through retry `continue` paths. + +## P2 — performance + +- **Closed 2026-07-30.** Each scheduling pass takes one atomic task/lease + snapshot, batches cached (TTL) reachability probes concurrently, and + evaluates candidate availability once. It no longer probes candidates or + scans active tasks once per queued task. +- **Closed 2026-07-30.** Active leases and per-harness, time-ordered quota + receipts are projection indexes. Availability uses indexed rolling-window + sums rather than decoding the event log; the task snapshot is a one-time + disposable compatibility cache rather than a full rewrite on every append. +- **Closed 2026-07-30.** `BenchmarkAssignPending{1K,10K}` and + `BenchmarkAppend{1K,10K}` report and enforce p95 budgets, with durable + fsync cost included in their respective paths. + +## Delivery order + +1. Durable event transitions + lease fencing. +2. Idempotent checkpoint/release/pickup transaction. +3. Worker-local rotation, completion, quota, and typed recovery. +4. Approval policy and performance indexes. +5. Only then: ingestion/UI expansion. + +## Release gate + +- **Pass:** build, vet, test, and race checks pass; unit/integration coverage + includes the defined fault and cross-machine cases. +- **Pending QA:** run the controlled soft, hard, milestone, thrash, + completion, and late-recovery paths on OpenCode, Claude, and Codex without + manual intervention for safe repository work. Follow the QA handoff above + and attach the resulting event ranges/artifacts before clearing this gate. + +## Capability restrictions at the agent boundary (2026-08-26) + +The rule now enforced: an agent may perform work and *request* lifecycle +changes; it may never perform one. + +- **New `authz.Agent` surface**, `GatedWrite`. `AuthorizeEvent` refuses every + event type Orchestra owns (`WorkPhaseChanged`, `ReviewRecorded`, + `TaskSubmitted`, `TaskCompleted`, `TaskLeased`, `ApprovalGranted`, …), and the + HTTP gate refuses their endpoints. Credential is `ORCHESTRA_AGENT_TOKEN`, the + only Orchestra credential that may enter an agent pane. +- **Gated surfaces could not reach the two endpoints written for them.** The + middleware admitted only paths ending in `/approval`, so `decision-request` + and `deferred` were 403 before their handlers ran, and the handlers' own + `AuthorizeEvent("ApprovalRequested")` was dead code. The allowlist is now + `authz.GatedWritePath`, and the gate keys off `CapabilityFor(s) == + GatedWrite` rather than naming MCP and Maven. +- **`/v1/harness/turn` was unreachable.** An unlabelled request defaults to the + Web surface, `sessions` is always non-nil, so every harness call returned 401 + in any deployment with web credentials. It is now exempt from the surface gate + and authenticates its own bearer token in the handler, like federation does. +- **`RequestHumanDecision` now requires ownership and fences on the lease.** It + blocked any task in any state, and its `TaskBlocked` event carried no + `harness_id`/`lease_epoch`. That is backwards twice over: an agent credential + was a way to block a queued task nobody was working on, and a question from + the session that *did* own the task was rejected by `Store.Append`'s fence. + Both fixed; six tests that had been exercising the unleased state now lease + first. + +Not built, deliberately: a `Capability` vocabulary, `CapabilitySet`, per-role +`SessionAuthority`, or role-to-capability project policy. One surface at one +capability level expresses "may ask, may not act", and the store already fences +every lifecycle event on `(harness_id, lease_epoch)`. Add roles when two agent +roles actually need different API rights — today the difference between an +implementer and a reviewer is what `agentctx` renders, not what the API allows. + +Still open, and it is the real enforcement: + +- **Execution capabilities are unenforced.** Network, secrets, paths outside + the worktree and destructive commands have no policy check; agents inherit + the harness pane's environment, and nothing in this repo sets it. `git push` + and a direct Gitea call succeed or fail purely on whatever credentials that + environment happens to hold. Credential isolation in the pane is an operator + task today, not a code path. +- **A reviewing session still cannot seal its own review**, and should not: + the worker seals on its behalf. That answers question 1 of the handoff. + +## Turn-boundary reconcile-failure escalation (2026-08-26) + +The gap the 2026-08-26 handoff deliberately left open. A failed reconcile at a +verified turn boundary was recorded and the turn continued, forever. One +failure is transient, so continuing is right. A streak means Orchestra can no +longer promise that the newest human input outranks the session's intent, and +continuing silently is exactly the failure shape this repo keeps producing. + +- `Coordinator.ReconcileFailureHandoff` (default 3, `defaultReconcileFailureHandoff`) + is the number of *consecutive* failures that escalate. Env: + `ORCHESTRA_RECONCILE_FAILURE_HANDOFF`. +- `Coordinator.noteReconcileResult` holds the streak per task, **keyed on the + lease epoch**. A successor never inherits its predecessor's count, so no + release path needs a cleanup hook. A success deletes the entry. The count is + written into `MonitorHealth.Sessions[id].LastError` as + `reconcile human input (N consecutive): ...`. +- On escalation `TurnDecision` asks for a handoff with reason + `reconcile_failure` and answers `prepare_handoff`. `RemoteTurn` answers the + same for a worker-owned session, on the same threshold. +- **The escalation only fires where rotation had no reason of its own.** It sits + on the `continue` branch, so an existing rotate, refuse, or prepare_handoff + keeps its own reason rather than having a second one manufactured for it. +- `reconcile_failure` is a real handoff reason: added to `continuity.reasons` + (or the artifact it produces would fail validation), to `herdr.handoffReason`, + and to the new `orchestrator.bypassReason` — which replaces the + `manual || milestone || thrash` comparison that was duplicated in `rotate` + and `TurnDecision`. Release therefore runs through the existing bypass path, + with no new state, blocker or recovery protocol. +- `CLIAdapter.RequestHandoffReason` explains the reason to the agent and says + explicitly that it is not a judgement about its work. +- **The worker ignored the coordinator's verdict entirely.** `federatedTurn` + read `answer.Decisions` and dropped `answer.Verdict` on the floor, so a + federated session could never be asked to hand off for any coordinator-side + reason. It now requests the handoff and records it on the session, which is + what the release loop watches for. + +Then the existing machinery does the useful part: the successor's +`Store.PreLease` reconcile fails closed while the source is down, so the task +waits in the queue instead of resuming from an older authority. + +Proofs: `internal/orchestrator/reconcile_escalation_test.go` (streak, reset, +rotation-wins, release-through-bypass, no-source, delivery-failure-is-not-a- +reconcile-failure, federated parity), `internal/integration/reconcile_escalation_test.go` +(the full loop: two turns continue, third hands off, successor refused while the +source is down, correction reconciled on recovery), and +`cmd/orchestra-worker/main_test.go:TestFederatedTurnActsOnPrepareHandoffVerdict`. + +Deviation from the requested shape: the threshold is one coordinator field, not +per-project `human_reconcile.turn_failure_handoff_after`. `TurnDecision` has no +project in scope, and `Soft`/`Hard`/`Thrash` are already coordinator-level for +the same reason. Making it per-project means plumbing the registry into the +coordinator, which is worth doing when a second project actually needs a +different number. + +## Burn-in instrumentation and readiness probe (2026-08-26) + +Feature work stops here. `BURNIN.md` is the runbook: evidence per run, the five +flows, the failure classification, and the pane credential cleanup. + +One addition, because the burn-in's main inspection was impossible without it: +`herdr.WriteLaunchContext` dumps the exact `agentctx.Build` result to +`/.orchestra/launch.md` at every launch, local (`Coordinator.Start`) +and federated (`orchestra-worker`). Reading it back from pane scrollback is not +equivalent, because the harness reflows and truncates. A write failure is +recorded, never fatal. Proof: +`internal/integration/reconcile_launch_test.go:TestLaunchWritesTheContextItSent` +compares the file against the instruction the adapter actually received. + +Probed live state, and the burn-in is blocked on deployment, not on code: + +- The API is up on homesrv (`/readyz` ready, gitea and jsonl configured). +- The workpc worker is up (pid 741, restarted 2026-08-26 11:39, no errors), + serving `workpc-claude` on a tmux backend and `workpc-opencode` on a herdr + unix socket. +- **The deployed worker is built from 97a9c65 (2026-07-30).** Every v3 unit from + the last two sessions is an uncommitted working-tree change, so neither the + container nor the worker has phases, review, submission, the agent surface, or + the reconcile escalation. Commit and redeploy both before any run. +- **Codex has no entry in `/etc/orchestra/harnesses.json`**, and no worker runs + on homesrv, so three of the four target harnesses cannot be exercised at all + today. +- Correction to an earlier assumption in this file: the live `config.jsonc` sets + no `backend` and no `address` on any of its six herdrs, so all six resolve to + `:9245` via `registry.defaultHerdrPort`. Both ports are closed, and + that is *not* evidence about the workpc harnesses, which use a tmux socket and + a unix socket. Federated reachability defers to worker heartbeat. Do not + diagnose harness availability from a TCP probe of 9245. + +## Burn-in deployment, 2026-08-26 18:35 + +Burn-in build identity: `6f9300b549362c4c5788f8845b56aaff9672d993`. The v3 stack +is committed (`7f12c7f`), then startup revision logging (`86b67d9`), then this +record. Both halves are built from `6f9300b` so the identity is HEAD, and a +later `deploy/build.sh` cannot silently produce a different revision than the +one deployed. + +- **Coordinator deployed.** Rebuilt on homesrv with `--build-arg + BUILD_REVISION/BUILD_TIME/BUILD_DIRTY`, container recreated, `/readyz` ready. + It now logs `orchestra revision 6f9300b... dirty false` at startup. +- **Worker staged, not installed.** `~/orchestra-deploy/orchestra-worker`, + sha256 `2b1c430...`. `install` and `systemctl restart` need root, which this + sandbox does not have, so the running worker is still the 2026-07-30 build. + Until it is installed the pair is mismatched and no task should be created. +- **Observability fixed before proceeding**, per the requirement that deployed + identity be evidence. Revision was previously visible only behind the operator + login, and the worker never logged its own build at all. Both now print it at + startup, so `docker logs orchestra-api` and `journalctl -u orchestra-worker` + are sufficient. +- **`deploy/build.sh`** stamps both binaries from one commit and refuses a dirty + tree, so a burn-in run cannot pair a new coordinator with an old worker. +- The deployed coordinator confirms the transport split directly: + `herdr workpc-opencode is worker-owned on workpc; coordinator probe skipped`, + while the three `homesrv-*` herdrs report `dial tcp 192.168.1.104:9245: + connect: connection refused`. + +Not done, and both need root: the `/etc/orchestra/worker.env` scrub (mode 0600, +root-owned) and its in-pane verification. No agent should run before that. + +## Pane environment and the opencode backend, probed 2026-08-26 18:45 + +Three findings, all blocking flow 1, none of them code defects in this session's +work. + +- **herdr is not running on workpc.** `herdr status server` reports `not + running`; the socket refuses connections and its log stops at 2026-07-30. + `workpc-opencode` cannot start a pane. Note that the worker logs `serving + harness workpc-opencode (opencode) on herdr backend` at startup **without + touching the socket**, so that line is not evidence of reachability. Same + shape as the older note about the coordinator never logging a herdr success: + absence of an error is not evidence here either. +- **The two workpc harnesses inherit different environments.** An opencode pane + is created by the herdr daemon and inherits *herdr's* environment, so scrubbing + `/etc/orchestra/worker.env` does not affect it. A claude pane comes from + `TmuxBackend.StartAgent`, which runs `tmux new-session` via + `exec.CommandContext` with no `Env` set, so the tmux server inherits the + worker's full environment and every pane under it does too. +- **The scrub alone cannot close the tmux path.** The worker needs + `ORCHESTRA_WORKER_TOKEN*` and `ORCHESTRA_FEDERATION_ADMIT_TOKEN` to function, + and the pane inherits exactly those. Closing it needs a filtered `cmd.Env` in + the backend, or a tmux server started separately with a clean environment. + Deliberately not built now: flow 1 is opencode only, and the burn-in order + puts claude at step 6. + +Sudo is not available in this sandbox, so the worker install, the restart, and +the `worker.env` scrub remain operator steps. + +## Browser operator database and UI refresh (2026-08-26) + +The browser login no longer depends on an operator copying a bcrypt hash into +deployment configuration. The live startup path in `cmd/orchestra/main.go` +opens `$ORCHESTRA_DATA/auth.db` through `internal/authn`, refuses to serve with +an empty operator database, and registers the database-backed session and +account handlers before wrapping the mux with `authz.HTTPWithSessions`. + +- `auth.db` is an embedded bbolt database created mode 0600. Passwords are + bcrypt-hashed before the record is written; login also performs bcrypt for an + unknown username to avoid an account-existence timing shortcut. +- `orchestra-user set -data DIR -username NAME` reads and confirms a password + from the terminal, creates the first operator, and resets an existing one. + The Docker API image includes this helper. The authenticated Settings screen + changes the current username/password and revokes every session for that + identity. +- An existing `ORCHESTRA_WEB_USERNAME`/`ORCHESTRA_WEB_PASSWORD_HASH` pair is + imported once if and only if the database has no users. Once a user exists, + those variables are ignored with an explicit startup log, so an old `.env` + cannot overwrite a database credential. +- The browser now gets its actual username from `GET /v1/ui/session`, renders + it in the shell, and has a dedicated account page. The login view was rebuilt + as a responsive desktop/mobile entry experience. +- Frontend state drift was fixed at the same time: `needs_attention` and + `in_review`, plus the three newer block reasons, are in the TypeScript model, + board lanes, status colors, diagnosis copy, and filtering. The seven-state + "All" board now has an explicit layout instead of falling back to one column. + +Verified from the working tree after rebuilding the embedded assets: +`go build ./...`, `go vet ./...`, and `go test ./...` all pass (21 test +packages). The frontend TypeScript build passes, all five API-client tests +pass, and Vite's production build emits the assets embedded by +`internal/webui`. diff --git a/BURNIN.md b/BURNIN.md new file mode 100644 index 0000000..5e40d97 --- /dev/null +++ b/BURNIN.md @@ -0,0 +1,3018 @@ +# Burn-in: live conformance, not more architecture + +Written 2026-08-26. The workflow is feature-complete enough to exercise. What +remains is empirical: run the same real task shape through each harness and +classify what breaks. Isolated tests are insufficient by design here, and only +the live owner path establishes conformance. + +Do not add workflow features while this is running. Findings decide the next +implementation work. + +## Burn-in build identity + +`77a2b323fabcf080d7542061ae2d7b3c34eef5b7` + +Both halves must report exactly this revision before a task is created. Neither +needs a credential now: the coordinator prints it in `docker logs orchestra-api` +and the worker in `journalctl -u orchestra-worker`. The same object is at +`GET /v1/admin/diagnostics` and `GET /v1/federation/workers` behind the operator +login. + +Build both with `deploy/build.sh `, which refuses a dirty tree. Later +documentation-only commits do not change this identity, so a rebuild either +passes this revision explicitly or accepts the new one and redeploys both +halves. Never one half. + +## Deployment state, 2026-08-26 18:35 + +| Half | State | +|---|---| +| Coordinator (homesrv) | **Deployed at 6f9300b.** Rebuilt with `--build-arg BUILD_REVISION`, recreated, `/readyz` ready, self-reporting the revision in its log | +| Worker (workpc) | **Staged, not installed.** `~/orchestra-deploy/orchestra-worker` sha256 `2b1c43071eaf6b5c15b110de39f204038a9629897e0ce3dacf45be96a6e6529e`. Installed binary is still the 2026-07-30 build | + +Two operator steps remain, both needing root: + +```sh +sudo install -m 0755 /home/kami/orchestra-deploy/orchestra-worker /usr/local/bin/orchestra-worker +sudo systemctl restart orchestra-worker +journalctl -u orchestra-worker -n 5 --no-pager # must print revision 6f9300b... +``` + +A third blocker, found while preparing the pane check: + +**herdr is not running on workpc.** `herdr status server` reports `not running`, +the socket at `/home/kami/.config/herdr/herdr.sock` refuses connections, and its +log stops at 2026-07-30. `workpc-opencode` therefore cannot start a pane, even +with the worker installed. The worker logs `serving harness workpc-opencode +(opencode) on herdr backend` at startup without touching the socket, so that +line is not evidence the backend is reachable. herdr is an interactive terminal +workspace manager: running `herdr` in a terminal launches or attaches to the +persistent session and starts the server. Confirm with `herdr status server` +before creating a task. + +Then scrub the pane environment before letting an agent run. + +### Where the pane environment actually comes from + +The two workpc harnesses inherit different environments, so one scrub does not +cover both. + +- **`workpc-opencode` (herdr backend).** The pane is created by the herdr + daemon, which is a separate long-running process. It inherits herdr's + environment, not the worker's. Scrubbing `worker.env` does nothing here. + Whatever environment herdr is started with is what every opencode pane gets. +- **`workpc-claude` (tmux backend).** `TmuxBackend.StartAgent` runs `tmux + new-session` through `exec.CommandContext` with no `Env` set + (`internal/herdr/tmux.go`). If the tmux server is not already up, the worker + starts it, and that server inherits the worker's full environment. Every + claude pane then inherits it too. + +This exposes a conflict the scrub alone cannot resolve. The worker legitimately +needs `ORCHESTRA_WORKER_TOKEN` (or `ORCHESTRA_WORKER_TOKEN_`) and +`ORCHESTRA_FEDERATION_ADMIT_TOKEN`, and deleting them breaks the worker. Keeping +them means the tmux path hands them to the agent. Closing it needs either a +filtered `cmd.Env` when the backend spawns a pane, or a tmux server started +separately with a clean environment. Not on flow 1's path, because flow 1 is +opencode only. Close it before step 6. + +## Baseline: one harness, one path + +`workpc-opencode` only. Codex is a coverage gap, not a prerequisite, and no +worker runs on homesrv. Prove one real path first: + +``` +real vikunja task +→ workpc-opencode +→ frame/research/plan +→ trajectory gate if configured +→ implement +→ independent review +→ task pr +→ human merge +→ completed +``` + +Then raise difficulty in this order: + +``` +1. opencode boring success +2. opencode mid-session human correction +3. opencode forced rotation +4. opencode pr rejection -> fix -> merge +5. opencode reconcile outage/recovery +6. representative cases on claude +7. add the codex adapter/config +8. cross-harness rotation: opencode -> claude/codex +``` + +## Herdr transports, so nobody misdiagnoses again + +No entry in the live `config.jsonc` sets `backend` or `address`, so all six +resolve to `:9245` (`registry.defaultHerdrPort`). Both ports are +closed. That says nothing about the workpc harnesses: they are worker-owned, +and the coordinator logs `worker-owned on workpc; coordinator probe skipped` +for each. The worker reaches `workpc-claude` over a tmux socket and +`workpc-opencode` over `/home/kami/.config/herdr/herdr.sock`. Do not diagnose +harness availability from a TCP probe of 9245. + +## Evidence to record per run + +One row per run. The launch instruction is now dumped to +`/.orchestra/launch.md` (`herdr.LaunchContextFile`) at every launch, +local and federated, so the context is auditable without reading pane +scrollback. + +``` +harness (claude | codex | opencode) +task id +lease epoch (every epoch, if the task rotates) +session ids (pane ids per epoch) +human decision ids +launch context (sha256 of .orchestra/launch.md, per launch) +handoff ref +git sha (at every boundary: launch, each handoff, review, submit, merge) +review ref +submission ref +pr id +completion receipt +``` + +## The five flows + +1. **Boring success.** `task → research → plan → implement → review → pr → merge`. +2. **Mid-session correction.** Agent is doing A, the human says B, the next + verified boundary delivers B, and A becomes history rather than a competing + instruction. +3. **Rotation.** Agent A hands off, agent B resumes. Switch harnesses across the + boundary if both are up. +4. **Human rejection.** Submit sha A, comment on the pull request, task reopens, + fix at sha B, a fresh review, the same pull request, merge. +5. **Failure path.** Human source goes down, the reconcile streak escalates, the + session hands off, the successor lease is refused, the source recovers, and a + corrected successor starts. + +## What to inspect by hand + +For every launch, read `.orchestra/launch.md` and ask only: + +``` +does this agent know +- what the task actually wants? +- what was most recently decided? +- what phase it is in? +- what earlier material is merely historical? +- what to do next? +``` + +Then compare that against what the model did. + +## Classify before fixing + +Every failure gets exactly one label before any code is written: + +``` +authority bug the wrong thing outranked the right thing +context-selection bug the renderer showed or hid the wrong material +lifecycle bug state, lease, phase or event handling is wrong +adapter/harness bug the pane, occupancy, boundary or prompt path is wrong +model-following failure the context was right and the model ignored it +operator-policy gap credentials, deployment or configuration +``` + +This matters because the cheap response to every failure is another prompt +rule, and prompt rules accumulated to compensate for lifecycle or adapter bugs +are how the enforcement boundary rots. A model-following failure is the only +class a prompt change should ever answer. + +## Pane credential cleanup (operator, in parallel) + +The agent pane inherits the worker's environment. On workpc the worker is +`orchestra-worker.service`, `User=kami`, `EnvironmentFile=/etc/orchestra/worker.env` +(mode 0600, root-owned, deliberately unreadable here). Remove from that file, and +from anything else the pane inherits: + +``` +gitea mutation token +vikunja mutation credentials +orchestra operator / system token +``` + +Retain only `ORCHESTRA_AGENT_TOKEN` and credentials a specific task genuinely +needs. If an agent needs a forge operation, it asks Orchestra to perform it. + +Then verify from inside a real pane: + +```bash +git push # must fail unless Orchestra supplied the auth +curl .../v1/tasks//submission # must be 403 for the agent surface +tea pr create # must have no usable mutation credential +``` + +Note what is *not* enforced by code: nothing in this repo scrubs the pane +environment, and a fake `git` earlier in `$PATH` is not enforcement because +`/usr/bin/git` bypasses it. Credential isolation is the enforcement. +Confinement of network, filesystem and destructive commands belongs to whatever +launches the process (herdr, a container, systemd, bubblewrap), not to +Orchestra, which supplies identity and policy inputs. + +## Exit criterion + +Roughly 10 to 20 real tasks, with the five flows covered on each harness that +is actually up. Then read the classified failures and decide the next +implementation unit from them. + +## Live findings, 2026-08-26 21:35 + +Both halves are paired at `6f9300b`, and herdr is up (0.7.5, protocol 17). What +remains before flow 1, and what the in-pane credential checks will really show. + +### Flow 1 has no repository to run in + +`workpc-opencode` declares exactly one project, `test-e2e`, pointing at +`/tmp/test-e2e` and `/tmp/test-e2e-worktrees`. Neither path exists on workpc +after today's reboot, and `/var/lib/orchestra/repos` does not exist there at +all. Nothing can be leased to this harness until it has a project backed by a +durable local repository. + +The only configured task source is Gitea `kami/correx` at +`https://gitea.kvmx.ru` (`ORCHESTRA_GITEA_URL/OWNER/REPO`), so a real task means +a `correx` issue. `correx` already has `machine_affinity: ["homesrv","workpc"]` +in the coordinator's `config.jsonc`. What is missing is a workpc-local entry: + +```jsonc +// /etc/orchestra/worker-projects.json, on workpc +{ + "correx": { + "repo": "/home/kami/orchestra/repos/correx.git", + "worktree_root": "/home/kami/orchestra/worktrees/correx", + "remote": "origin" + } +} +``` + +Not `/tmp`. The burn-in outlives a reboot. + +### There is no vikunja ingest + +This repo has no Vikunja provider. `/readyz` reports `gitea` and `jsonl`. A +"real vikunja task" cannot enter Orchestra today: it arrives as a Gitea issue or +through the JSONL watcher. + +### `git push` from a pane will succeed, and no Orchestra setting stops it + +Corrected 21:45. The mechanism is SSH, not HTTPS. HTTPS has no stored +credential on workpc, where `git push` over `https://` fails with `could not +read Username`. Pushing works over `ssh://git@gitea.kvmx.ru:2222` using kami's +default RSA identity, which Gitea lists as the key named `workpc`. It carries no +passphrase, so no agent is needed. + +That identity is what lets the *worker* push, and panes run as the same user +with the same home directory, so an agent inherits it. `worker.env` is +irrelevant to this. Real isolation needs panes under a different unix user. +Expect `git push --dry-run` to succeed from a pane, and record it as an +operator-policy gap rather than a code defect. + +The worker inherits ambient Git credentials by design: `git()` in +`cmd/orchestra-worker/main.go` runs `exec.CommandContext` with no environment of +its own. There is no separate credential path for the worker to hold something +the pane does not. + +### The agent surface is currently unauthenticated + +`ORCHESTRA_AGENT_TOKEN` is unset on the coordinator, and an unset surface token +means the middleware performs no check for that surface. Probed live: + +``` +POST /v1/tasks//decision-request -H 'X-Orchestra-Surface: agent' -> 404 (reached the handler) +POST /v1/tasks//phase -H 'X-Orchestra-Surface: agent' -> 403 (refused) +``` + +The capability boundary holds. Authentication does not. Set +`ORCHESTRA_AGENT_TOKEN` in the coordinator `.env` before the burn-in, and note +the same is true of `ORCHESTRA_MCP_TOKEN` and `ORCHESTRA_MAVEN_TOKEN`, both +unset. + +### Two more unset settings that change burn-in behaviour + +- `ORCHESTRA_FEDERATION_ADMIT_TOKEN` is unset, so any caller may register a new + worker identity. Existing identities stay protected, because registering an + existing id with a different token is refused. +- `ORCHESTRA_REVIEW_ACTORS` is unset, which `human.Trust` reads as "anyone not + explicitly ignored". Flow 4 will accept a task-moving comment from any Gitea + actor. Set it to `kami`. + +## Flow 1 setup, 2026-08-26 21:45 + +Target: `kami/test-e2e` on Gitea, a throwaway repo, rather than `correx`. + +Done: + +- **Ingest switched.** `ORCHESTRA_GITEA_REPO=test-e2e` in the coordinator `.env` + (previous file kept as `.env.pre-burnin-20260826`), container recreated, still + reporting `6f9300b`. The project id must equal the repo name, because + `main.go` sets `Project: os.Getenv("ORCHESTRA_GITEA_REPO")`. `test-e2e` + already exists in `config.jsonc` with `machine_affinity: ["workpc"]`. +- **Durable repo on workpc.** Bare clone at + `/home/kami/orchestra/repos/test-e2e.git`, worktree root + `/home/kami/orchestra/worktrees/test-e2e`. Origin points at + `ssh://git@gitea.kvmx.ru:2222/kami/test-e2e.git`, and `git push --dry-run` + reports `Everything up-to-date`. The old `/tmp` paths did not survive the + reboot and must not come back. + +Remaining, needs root: + +```jsonc +// /etc/orchestra/worker-projects.json +{ + "test-e2e": { + "repo": "/home/kami/orchestra/repos/test-e2e.git", + "worktree_root": "/home/kami/orchestra/worktrees/test-e2e", + "remote": "origin" + } +} +``` + +Then `sudo systemctl restart orchestra-worker`. + +Store state, checked before the first run: 30 tasks, all `test-e2e`, with 23 +blocked, 6 completed and 1 failed. All are July leftovers and none holds a +lease. The coordinator has run `ResumeAnsweredBlockers` every second for hours +without resuming any of them, so they are inert rather than merely quiet. The +July stuck task `06FT6CKD9Y98AZRX6X8K3QXFZG` is now `failed`. + +A stale branch `orchestra/scratch/oc-06ftgkjadcd2hwjn2zwjen90q4` exists on the +remote from an earlier run. Harmless, but it is not from this burn-in. + +## Run 1: task 06G3YR34117MAYT6KEAC9RJHD0, 2026-08-26 + +Evidence ledger: + +``` +harness workpc-opencode (herdr backend) +task 06G3YR34117MAYT6KEAC9RJHD0 +source gitea:test-e2e/1 "Add a --version flag to the healthcheck script" +attempts 3 leases: two nacked, third launched +pane wN:p1, agent oc-06g3yr34117mayt6keac9rjhd0 +agent session ses_fc096de31ffelZeMngQEyBTxfh +worktree /tmp/test-e2e-worktrees/06G3YR34117MAYT6KEAC9RJHD0 +branch orchestra/06G3YR34117MAYT6KEAC9RJHD0 +head at launch 5e6c4783d4213a934dc486160b888b6014bc2d03 +launch context .orchestra/launch.md, 2138 bytes +phase frame +decisions none +``` + +### What worked + +- **The launch context is right.** Authority order, the ambiguity ladder, the + phase brief with `frame: ... Do not change code`, `Current human decisions: + None recorded`, and a verified git state with worktree, branch and head. +- **The agent respected the phase.** `agent_status: done`, no code changes, only + Orchestra's own scratch directory in the tree. +- **A bad launch nacked cleanly.** The first attempt failed on the base checkout + and the task went back to `queued` with `lifecycle_phase: launch_nacked` and + `attempt: 1`. No orphan pane, no stuck lease. + +### Fixed during the run + +- **F1, authority bug, `2753a8d`.** Every federated launch died with `effective + intent: federation: 401 Unauthorized: unauthorized surface`. + `GET /v1/tasks//intent` exists for workers, and the authz worker-path + exemption never included it. Twenty test packages passed throughout. +- **F2, authority bug, `09e572f`.** The rendered goal was the issue title alone + and acceptance read `Not stated.`, because `Gitea.event` parsed the issue body + and dropped it from the `TaskCreated` payload. The store already read + `description`. Every Gitea-sourced task so far ran on its title. +- **F3, self-inflicted, `a0209a2`.** The launch dump left `.orchestra/` + untracked in a worktree whose own context said `uncommitted changes: false`. + It would have polluted the gate, the review diff and the agent's `git status`. + +### Open, classified + +- **F4, adapter/harness.** `rotation ... activity degraded: activity unknown: + adapter: resolve session file: adapter: harness "opencode" has no + session-file resolver`. Thrash and activity triggers are permanently degraded + on opencode. Observable in the worker's `last_error`, non-fatal. +- **F5, lifecycle.** The router never leased this task. Every gate checks out by + hand: worker `online: true`, `herdr_status: reachable`, fresh `checked_at`, + `supported_projects: ["test-e2e"]`, quota empty, no active leases, capability + empty. A direct `POST /v1/tasks//lease` succeeded instantly. The refusal + is upstream of `Store.Lease` and silent by construction + (`internal/router/router.go:197`). All three runs were leased by hand. +- **F6, observability.** Worker health reports `active_task: null` while the + store shows the task leased and herdr shows a live opencode agent in the + task's worktree. +- **F7, operator policy.** `ORCHESTRA_TUI_TOKEN` is unset, and an unset surface + token means no authentication for that surface. That is how the manual leases + above were issued, unauthenticated, from another machine. The TUI surface is + `FullControl`, so this is the whole control plane, not just the two agent + request endpoints. Set the token. +- **F8, correctness.** `human.Reconciler.Reconcile` iterates every configured + source for every task, so the three queued `correx` tasks have their external + ids looked up in `kami/test-e2e`. A source should only reconcile the tasks + that came from it. Likely why those three never lease. + +## Fix pass before run 2, 2026-08-26 23:50 + +Burn-in identity is now `77a2b32`. The coordinator is deployed at it. The worker +is staged at it, sha256 `2a850f2d1102d390...`, still needing root to install. + +### Closed + +- **F7, security.** `authz.RequireCredentials` refuses startup when a + full-control surface has no token, rather than logging it. + `ORCHESTRA_TUI_TOKEN` is set in the coordinator `.env`. Verified live: an + unauthenticated `POST` on the TUI surface now returns 401. Web is exempt + because `Sessions` makes its login mandatory. `ORCHESTRA_MCP_TOKEN`, + `ORCHESTRA_MAVEN_TOKEN` and `ORCHESTRA_AGENT_TOKEN` remain unset, so those + surfaces are still unauthenticated for reads and for their three request + endpoints. Bounded by capability, worth closing, not startup-fatal. +- **F5, lifecycle.** Every eligibility gate now records a `router.Rejection`, + exposed at `GET /v1/router/health` and reset per pass. No gate was weakened. + The live output immediately explained the three stuck `correx` tasks: + `worker has not declared project correx`. A queued task in retry backoff was + skipped before the candidate loop and recorded nothing at all, which is the + shape that hid the original case; it now reports + `retry backoff until