v3 workflow: intent, phases, review, submission, enforcement, burn-in

The v3 stack, previously an uncommitted working tree, plus this session's two
units and the burn-in instrument. This commit is the burn-in build identity:
coordinator and worker must both report this revision before a task is created.

Workflow (earlier sessions, uncommitted until now): human decision events and
reduction, source cursors and reconcile-before-launch, turn-boundary
reconciliation, internal/agentctx as the single renderer, ace-fca phases with
sealed artifacts, the trajectory gate, bounded grilling, independent review,
task pr enforcement, and human review reflection.

Capability restrictions at the agent boundary: an authz.Agent surface at
GatedWrite may ask and may not act. It also fixes two bugs the unit exposed --
gated surfaces could not reach the two endpoints written for them, and
RequestHumanDecision would block an unowned task while rejecting a question
from the session that did own it.

Turn-boundary reconcile-failure escalation: a streak of consecutive failures
asks the session to hand off, fenced on the lease epoch, with reconcile_failure
as a real handoff reason. The worker was dropping the coordinator's verdict on
the floor; it now acts on it.

Burn-in: herdr.WriteLaunchContext dumps the exact agentctx.Build result to
<worktree>/.orchestra/launch.md at every launch, local and federated. BURNIN.md
is the runbook. deploy/build.sh stamps both binaries from one commit.

go build, go vet and go test ./... pass, 20 packages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-26 18:31:20 +04:00
parent 97a9c65302
commit 7f12c7fc37
78 changed files with 16417 additions and 352 deletions
+165
View File
@@ -1,5 +1,36 @@
# Orchestra audit — handoff first
## 2026-08-11 — tmux backend and Claude in-place context rollover
- `orchestra-worker` now selects `ORCHESTRA_WORKER_BACKEND=herdr|tmux`.
The tmux backend is intentionally Claude-only: it creates one detached,
isolated tmux session per task, resolves configured tmux pane indexes rather
than assuming `0.0`, starts Claude in the task worktree, handles the exact
workspace-trust prompt, captures the pane, forwards explicit input, reports
health/status, and retains/kills the session at the existing lifecycle
boundaries. Codex and OpenCode still use herdr.
- Registry entries accept `"backend":"tmux"`. Those entries are worker-owned
even on the coordinator machine, so the Dockerized coordinator neither
probes nor tries to operate a host tmux socket. Their router health remains
gated by a fresh worker heartbeat and local backend check.
- Claude rotation no longer asks the agent to duplicate the workpc hook's
handoff in `.orchestra-handoff-report.md`. A changed `HANDOFF.md`, observed
after Claude is idle, drives the persisted in-place sequence `/clear` +
`ENTER`, then `@HANDOFF.md` + `ENTER`; the exhausted transcript identity is
discarded so the next observation resolves the new session. The canonical
cross-worker release protocol remains separate and unchanged.
- The portable workpc Claude configuration (`settings*.json`, `CLAUDE.md`,
status line, hooks, and helper scripts) was checksum-verified after copying
to homesrv. The prior homesrv files are backed up under
`/home/kami/.claude/backups/workpc-import-20260811T120000`; machine/session
state was excluded and local-only hooks were preserved.
- Verified in the source tree with the real isolated-tmux lifecycle test,
focused backend/worker tests, `go build ./...`, `go vet ./...`,
`go test ./...`, `go test -race ./...`, and the web lint/test/production
build. No worker binary, registry file, service, container, or live pane was
deployed/restarted/touched; live Claude/tmux behavior remains a deployment
QA item rather than a completed claim.
Audited 2026-07-30 against the working tree, spec, deployed coordinator,
workpc worker, event log, and live herdr (read-only).
@@ -168,3 +199,137 @@ verified live, which is why deleting it costs nothing.
completion, and late-recovery paths on OpenCode, Claude, and Codex without
manual intervention for safe repository work. Follow the QA handoff above
and attach the resulting event ranges/artifacts before clearing this gate.
## Capability restrictions at the agent boundary (2026-08-26)
The rule now enforced: an agent may perform work and *request* lifecycle
changes; it may never perform one.
- **New `authz.Agent` surface**, `GatedWrite`. `AuthorizeEvent` refuses every
event type Orchestra owns (`WorkPhaseChanged`, `ReviewRecorded`,
`TaskSubmitted`, `TaskCompleted`, `TaskLeased`, `ApprovalGranted`, …), and the
HTTP gate refuses their endpoints. Credential is `ORCHESTRA_AGENT_TOKEN`, the
only Orchestra credential that may enter an agent pane.
- **Gated surfaces could not reach the two endpoints written for them.** The
middleware admitted only paths ending in `/approval`, so `decision-request`
and `deferred` were 403 before their handlers ran, and the handlers' own
`AuthorizeEvent("ApprovalRequested")` was dead code. The allowlist is now
`authz.GatedWritePath`, and the gate keys off `CapabilityFor(s) ==
GatedWrite` rather than naming MCP and Maven.
- **`/v1/harness/turn` was unreachable.** An unlabelled request defaults to the
Web surface, `sessions` is always non-nil, so every harness call returned 401
in any deployment with web credentials. It is now exempt from the surface gate
and authenticates its own bearer token in the handler, like federation does.
- **`RequestHumanDecision` now requires ownership and fences on the lease.** It
blocked any task in any state, and its `TaskBlocked` event carried no
`harness_id`/`lease_epoch`. That is backwards twice over: an agent credential
was a way to block a queued task nobody was working on, and a question from
the session that *did* own the task was rejected by `Store.Append`'s fence.
Both fixed; six tests that had been exercising the unleased state now lease
first.
Not built, deliberately: a `Capability` vocabulary, `CapabilitySet`, per-role
`SessionAuthority`, or role-to-capability project policy. One surface at one
capability level expresses "may ask, may not act", and the store already fences
every lifecycle event on `(harness_id, lease_epoch)`. Add roles when two agent
roles actually need different API rights — today the difference between an
implementer and a reviewer is what `agentctx` renders, not what the API allows.
Still open, and it is the real enforcement:
- **Execution capabilities are unenforced.** Network, secrets, paths outside
the worktree and destructive commands have no policy check; agents inherit
the harness pane's environment, and nothing in this repo sets it. `git push`
and a direct Gitea call succeed or fail purely on whatever credentials that
environment happens to hold. Credential isolation in the pane is an operator
task today, not a code path.
- **A reviewing session still cannot seal its own review**, and should not:
the worker seals on its behalf. That answers question 1 of the handoff.
## Turn-boundary reconcile-failure escalation (2026-08-26)
The gap the 2026-08-26 handoff deliberately left open. A failed reconcile at a
verified turn boundary was recorded and the turn continued, forever. One
failure is transient, so continuing is right. A streak means Orchestra can no
longer promise that the newest human input outranks the session's intent, and
continuing silently is exactly the failure shape this repo keeps producing.
- `Coordinator.ReconcileFailureHandoff` (default 3, `defaultReconcileFailureHandoff`)
is the number of *consecutive* failures that escalate. Env:
`ORCHESTRA_RECONCILE_FAILURE_HANDOFF`.
- `Coordinator.noteReconcileResult` holds the streak per task, **keyed on the
lease epoch**. A successor never inherits its predecessor's count, so no
release path needs a cleanup hook. A success deletes the entry. The count is
written into `MonitorHealth.Sessions[id].LastError` as
`reconcile human input (N consecutive): ...`.
- On escalation `TurnDecision` asks for a handoff with reason
`reconcile_failure` and answers `prepare_handoff`. `RemoteTurn` answers the
same for a worker-owned session, on the same threshold.
- **The escalation only fires where rotation had no reason of its own.** It sits
on the `continue` branch, so an existing rotate, refuse, or prepare_handoff
keeps its own reason rather than having a second one manufactured for it.
- `reconcile_failure` is a real handoff reason: added to `continuity.reasons`
(or the artifact it produces would fail validation), to `herdr.handoffReason`,
and to the new `orchestrator.bypassReason` — which replaces the
`manual || milestone || thrash` comparison that was duplicated in `rotate`
and `TurnDecision`. Release therefore runs through the existing bypass path,
with no new state, blocker or recovery protocol.
- `CLIAdapter.RequestHandoffReason` explains the reason to the agent and says
explicitly that it is not a judgement about its work.
- **The worker ignored the coordinator's verdict entirely.** `federatedTurn`
read `answer.Decisions` and dropped `answer.Verdict` on the floor, so a
federated session could never be asked to hand off for any coordinator-side
reason. It now requests the handoff and records it on the session, which is
what the release loop watches for.
Then the existing machinery does the useful part: the successor's
`Store.PreLease` reconcile fails closed while the source is down, so the task
waits in the queue instead of resuming from an older authority.
Proofs: `internal/orchestrator/reconcile_escalation_test.go` (streak, reset,
rotation-wins, release-through-bypass, no-source, delivery-failure-is-not-a-
reconcile-failure, federated parity), `internal/integration/reconcile_escalation_test.go`
(the full loop: two turns continue, third hands off, successor refused while the
source is down, correction reconciled on recovery), and
`cmd/orchestra-worker/main_test.go:TestFederatedTurnActsOnPrepareHandoffVerdict`.
Deviation from the requested shape: the threshold is one coordinator field, not
per-project `human_reconcile.turn_failure_handoff_after`. `TurnDecision` has no
project in scope, and `Soft`/`Hard`/`Thrash` are already coordinator-level for
the same reason. Making it per-project means plumbing the registry into the
coordinator, which is worth doing when a second project actually needs a
different number.
## Burn-in instrumentation and readiness probe (2026-08-26)
Feature work stops here. `BURNIN.md` is the runbook: evidence per run, the five
flows, the failure classification, and the pane credential cleanup.
One addition, because the burn-in's main inspection was impossible without it:
`herdr.WriteLaunchContext` dumps the exact `agentctx.Build` result to
`<worktree>/.orchestra/launch.md` at every launch, local (`Coordinator.Start`)
and federated (`orchestra-worker`). Reading it back from pane scrollback is not
equivalent, because the harness reflows and truncates. A write failure is
recorded, never fatal. Proof:
`internal/integration/reconcile_launch_test.go:TestLaunchWritesTheContextItSent`
compares the file against the instruction the adapter actually received.
Probed live state, and the burn-in is blocked on deployment, not on code:
- The API is up on homesrv (`/readyz` ready, gitea and jsonl configured).
- The workpc worker is up (pid 741, restarted 2026-08-26 11:39, no errors),
serving `workpc-claude` on a tmux backend and `workpc-opencode` on a herdr
unix socket.
- **The deployed worker is built from 97a9c65 (2026-07-30).** Every v3 unit from
the last two sessions is an uncommitted working-tree change, so neither the
container nor the worker has phases, review, submission, the agent surface, or
the reconcile escalation. Commit and redeploy both before any run.
- **Codex has no entry in `/etc/orchestra/harnesses.json`**, and no worker runs
on homesrv, so three of the four target harnesses cannot be exercised at all
today.
- Correction to an earlier assumption in this file: the live `config.jsonc` sets
no `backend` and no `address` on any of its six herdrs, so all six resolve to
`<machine>:9245` via `registry.defaultHerdrPort`. Both ports are closed, and
that is *not* evidence about the workpc harnesses, which use a tmux socket and
a unix socket. Federated reachability defers to worker heartbeat. Do not
diagnose harness availability from a TCP probe of 9245.