Slice B, second half. OperatorInterventionRecorded is the one command for
saying "I fixed this by hand": a manual repair happens outside Orchestra by
definition, so the only honest way to have the evidence is for the person
who made it to state it. Inferring "an operator probably intervened" from a
gap would put guesses into the record the ledger is built from.
The debt projection now consumes both new kinds. A closed incident is one
observation carrying its repeat count as intensity, so recurrence stays a
count of independent incidents: 301 repeats on one lease and 2 on another
is a recurrence of two with an intensity of 303, not a recurrence of 303.
Both kinds were previously reported as holes in the system. They are
ordinary evidence now, so their absence from a history is a fact about that
history, and the gap list says so.
The worker also stamps a per-process incarnation on registration and every
heartbeat. Nothing else on the wire distinguishes a restarted worker from a
running one, and an incident cannot outlive the process that reported it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F18. A single last_error slot destroyed causal evidence twice. Run 7
kept only the last of four failures. In run 11 a five-second retry loop
on a dead task pinned the slot for twenty-six minutes, so the live
task's own expiry was never visible at all, and run 12 lost diagnosis
time to the same thing before F58 removed the flood.
WorkerHealth now carries up to sixteen distinct observations, each with
its repeat count and first/last times. Collapsing is by message rather
than by position, because a loop interleaved with other failures would
otherwise still flush the ring. Eviction drops the least recently seen.
last_error and error_at keep their wire names and still report only the
newest failure, so nothing reading them has to change.
The ring lives in memory beside last_error and is not persisted, which
is the behaviour last_error already had across a restart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
An implementer that finds the plan contradicted by the code had two options,
both bad: work around it silently, or improvise a different plan inside the
phase meant to execute one. PlanMismatch is the third.
The report carries an observation and nothing else. It may not propose a
replacement plan, because writing the next plan is the planning phase's work.
requested_action stays advisory: replan, research, or human_decision is a
recommendation, and Orchestra decides.
Staleness is checked before anything is recorded. A report names the plan ref
and the commit it was written against, both filled by the worker from what it
can verify rather than from what the agent asserted. A report against an older
plan says nothing about the current one, and one against an older tree may
already be fixed. Neither is replayed.
The reducer keeps two things apart that are easy to conflate:
mismatch recorded != plan superseded
A plan stops being accepted only when a replacement is actually sealed, so an
abandoned replan leaves the accepted plan and its verified progress intact. On
a real re-seal the old ref moves to PlanHistory and its progress stops counting,
while the verification events stay in the log as provenance.
human_decision never reopens. It blocks with a packet stating what was observed
and what it contradicts, and a human answer can resolve the contradiction
without resealing anything: the plan, its progress and the phase all survive,
and the answer outranks the plan where they differ. Turning every ambiguity
into a replan would put the planner above the person who set the goal.
The backward edge is Orchestra's alone. CanReopenPhase is separate from
CanTransitionPhase, which every path validating an agent's request uses, so
phase-request.json still refuses a move back. An agent asks by reporting a
mismatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A detailed plan that nothing enforces is a document. This makes the phases
executable: the implementer may write exactly one status, and every other
status is a conclusion Orchestra reaches by running the plan's own commands.
agent may request: ready_for_verification
agent may not assert: verified, awaiting_manual_verification, failed, skipped
The worker resolves commands from the coordinator, never from the request, so a
request cannot smuggle in a command the planner did not write. They run as argv
through exec with Dir set to the worktree, which is the quality gate's existing
envelope and not a weaker one. There is no shell, so a pipe is a literal
argument.
Project policy decides executable reach. registry.Project.Verification matches
argv positionally, and an absent policy refuses everything: a plan command is
agent-authored, so inheriting the operator-authored gate's reach by default
would be the wrong direction to fail in. A refused command is refused before
anything runs, and the refusal names the project and the command so the planner
learns its real reach.
Two bindings make the record mean something later. PlanRef, so progress earned
under plan A cannot survive into plan B. AtSHA, so "verified" does not outlive
the code that made it true: a record whose commit has moved is retained as
provenance and rendered as stale, never as a claim about the current tree.
Both are the same failure this codebase already fixed for reviews, which bind
to the commit they examined.
Manual steps hold a phase at awaiting_manual_verification. The sign-off is an
ordinary human decision whose subject carries the plan ref and the phase id, so
a later "looks good" on an unrelated thread cannot satisfy a gate nobody was
discussing.
A plan sealed before plan.md declares no executable unit, and says so: the
implement context states that phase progress is unavailable and the work
continues under the old semantics. Inventing phases it never had would be worse
than admitting it has none.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The completion tail ended at TaskCompleted with no pull request. Nothing in
the running system ever called the review or submission endpoints: the whole
event log holds zero ReviewRecorded and zero TaskSubmitted, so the merge
reflection, the publisher and the human trust boundary had no entry point.
Four links, in the order the tail needs them:
- finalize commits first and runs the quality gate against the committed
tree, so the gate result is bound to the commit being submitted.
CheckSubmission requires gate sha, review sha and head sha to be one
commit, which a gate run on the pre-commit tree can never satisfy.
- The worker seals the reviewer's findings and submits, through a new
/v1/federation/workers/<id>/submit. A blocking review returns the task to
implementation instead; a project with no forge still completes directly.
- The reviewing session is told where findings go. The brief asked for
findings and named no file, and it described a diff nobody supplied.
- GiteaPublisher.Push asks the forge what the branch holds before reaching
for a local checkout. A worker-owned worktree is on another machine and
has already pushed the commit; the coordinator has no such directory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Two gaps F33 left behind.
F34: the renewal gate exempts a lease with no baseline, so a pane that opened
and never started got a full free renewal period. That is the exact case the
gate exists to catch, and it happened live at 01:33:37: the stuck run 5 lease
renewed to 22:03 on a pane that had not moved since 01:13. Baseline the
progress hash at launch, where the pane is already being read.
F35: nothing could re-poke a live pane. Orchestra can put text in an editor and
be wrong about whether it landed, and the only recovery was to destroy the
lease and wait out expiry, roughly an hour. The new "resubmit" action presses
Enter on text Orchestra itself submitted. It changes no lifecycle state so it
appends no event, and it is fenced like an approval: a live worker-owned pane
at the capture revision the operator was looking at.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
F25. rotationTick returned early for claude before reaching federatedTurn,
which has one call site below that return. On the harness both burn-in runs
used, no phase request could ever be read and every human decision recorded
against a live session went undelivered. Claude still skips the occupancy
state machine below, because it owns its context rollover through the
installed hook. A turn boundary is not a rotation.
F26. The phase brief listed every domain-legal target, so run 4's frame
session read "research, implement" and asked for implement, which the
project's path refuses. The path is Orchestra's to know: the brief now names
one step and says a wrong target comes back with the right one.
F27. A refused request only reached recordError, leaving the agent to rewrite
the same rejected file forever with nothing telling it why. federation.
StatusError makes a 409 classifiable, and the refusal is delivered through
sendPrompt under the F20 guarantee. A transport failure is not an answer: the
request survives and the agent is told nothing.
The F25 regression test fails against the unfixed rotationTick.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
The brief told the agent to ask for a phase change and never carried the
asking. The agent asked in prose, no code represented the request, and the
session idled until its lease expired. That is what failed run 3.
F21. The agent asks with .orchestra/phase-request.json, and seals
research.json or plan.json where the phase it is leaving produces one. At a
verified turn boundary the worker checks the phase belief, the transition and
the artifact, then calls the coordinator with its lease epoch and a derived
operation id. AdvanceWorkPhase is unchanged, so a request cannot reach a move
the operator surface could not also make. Redelivery is idempotent.
F22. A session now records the phase it was launched to run. One that no
longer matches its task rotates with reason phase_changed, whether this worker
asked for the change or an operator made it.
F20. CLIAdapter.prompt sent handoff and rotation prompts without confirming
them, which is the failure F20 exists to catch. Fixed at the shared call site.
F23 needed no change. Issue comments already become decisions with no
submission, through Reconciler.Reconcile at PreLease and at every turn
boundary. The earlier finding searched internal/operations alone and was
wrong. Tests now cover the boundary it turns on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Burn-in run 2 ingested its task and then sat queued forever. Every herdr in
the live config declares quota_limit_5h and quota_limit_weekly, the event log
holds zero QuotaReported events, and QuotaSince reported an empty window as
unknown. QuotaAvailability fails closed on unknown, so no harness could ever
be leased, and the only producer of a receipt is a completed lease.
The event log is Orchestra's whole accounting source, so a window holding no
receipts is observable zero consumption. QuotaSince now reports known for an
empty window and for a harness that has never reported. A receipt that
declares its own consumption unknown still fails closed.
The refusal also lied about its cause. federatedAvailability collapsed a base
gate refusal into the federation health string, so router health said "stale
heartbeat or unhealthy local backend" while the heartbeat was one second old.
Availability gates now name themselves through an optional
ReasonedAvailability contract: quota refusals say whether usage is unknown or
the window is exhausted and by how much, and worker refusals distinguish an
unregistered worker, a never-probed backend, a stale heartbeat, a stale
health check, and an unreachable backend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v3 stack, previously an uncommitted working tree, plus this session's two
units and the burn-in instrument. This commit is the burn-in build identity:
coordinator and worker must both report this revision before a task is created.
Workflow (earlier sessions, uncommitted until now): human decision events and
reduction, source cursors and reconcile-before-launch, turn-boundary
reconciliation, internal/agentctx as the single renderer, ace-fca phases with
sealed artifacts, the trajectory gate, bounded grilling, independent review,
task pr enforcement, and human review reflection.
Capability restrictions at the agent boundary: an authz.Agent surface at
GatedWrite may ask and may not act. It also fixes two bugs the unit exposed --
gated surfaces could not reach the two endpoints written for them, and
RequestHumanDecision would block an unowned task while rejecting a question
from the session that did own it.
Turn-boundary reconcile-failure escalation: a streak of consecutive failures
asks the session to hand off, fenced on the lease epoch, with reconcile_failure
as a real handoff reason. The worker was dropping the coordinator's verdict on
the floor; it now acts on it.
Burn-in: herdr.WriteLaunchContext dumps the exact agentctx.Build result to
<worktree>/.orchestra/launch.md at every launch, local and federated. BURNIN.md
is the runbook. deploy/build.sh stamps both binaries from one commit.
go build, go vet and go test ./... pass, 20 packages.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All five defects filed while implementing B18, plus the router flake that
predated them. None of this has run on the deployed instance: the service
is stopped and /usr/local/bin/orchestra predates every change here.
B20 is the one that could silently defeat approvals. The capture revision
was UnixNano, so it changed on every read and said nothing about whether
the pane had changed; it is now an FNV-1a hash of the pane text, changing
iff the text does. The worse half was precedence: capture() preferred the
coordinator over a published worker capture, handing Queue a timestamp the
owning worker's staleness check could never match, so every federated
approval resolved "stale" and the keystroke never happened. Worker captures
now win — their existence means a registered worker owns that pane — and
capturePane follows the same precedence via Capture.Source rather than
guessing.
B19 was filed as "federated approvals emit no event", which overstated it:
the resolution half already existed, and correctly fires only on an
acknowledged worker report. The missing half was the request. Server.action
now appends ApprovalRequested at queue time, subject_ref set to the command
ID the later resolution carries. If that append fails the queued command is
resolved "rejected" — a keystroke that left no audit trail must not run.
B21 bounds the command list: resolved commands prune after 30 minutes on
both Queue and Commands, pending ones never at any age, since dropping one
would discard an operator decision. The persistence half stays open and is
recorded as such — captures and commands are still in-memory only.
S12 splits ORCHESTRA_NTFY_TOKEN, which was both the secret handed to the
ntfy server and a valid inbound credential for the ntfy surface; the latter
is now ORCHESTRA_NTFY_SURFACE_TOKEN. Breaking: a deployment relying on the
old dual use has no inbound gate until it sets the new variable. S13
deletes the dead auth() copy of the authorization policy.
The router flake was in the test, not in assignment. Store.Tasks() ranges a
map, and the assertion indexed two separate Tasks() calls, failing whenever
the orderings disagreed; instrumenting it showed a valid TaskLeased and a
genuinely leased task on every "failing" run. It now snapshots once and
asserts that exactly one task is leased, and passes at -count=60.
AUDIT.md records what is still not done: the deployed env and binary, the
live re-verification B13-B17 has always lacked, and two operational faults
found in the journal that block it — all six herdrs are refusing
connections, and ntfy delivery is failing 403 on every send.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
Introduces the browser-facing surface and the worker-side protocol that
backs it:
- internal/ui: joined read model plus per-task lifecycle and approval
controls, kept separate from the raw endpoints workers and harnesses
depend on.
- internal/webui + web/: Vite/React app, build output embedded via
go:embed and served as an SPA fallback.
- federation: per-(worker, task) captures with a monotonic revision that
advances only when pane text actually changes, and a command queue
restricted to grant_approval / deny_approval, each bound to the capture
revision the operator acted on.
- orchestra-worker: publishes captures and executes commands only after
re-reading the pane and confirming the revision still matches. Sends
keystrokes only for a visible y/n prompt or OpenCode's fully labelled
selector, and refuses to deny through that selector rather than guess
at unobservable navigation.
This is the ownership boundary AUDIT.md's B14 and B17 call for: approval
becomes an explicit, revision-bound operation executed by the worker that
owns the pane, instead of a side effect of prompting over a
coordinator-driven remote socket.
Also ignores the web build inputs and outputs. node_modules ships vendored
Go packages, so go build and go test walk into it if it is merely
untracked; both node_modules and .node_modules are excluded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
Register() previously trusted a self-declared id and self-chosen token
from any caller, and let a second caller silently hijack an existing
worker id by re-registering it with a different token. Adds an optional
pre-shared AdmitToken (ORCHESTRA_FEDERATION_ADMIT_TOKEN) and requires a
same-id re-registration to present the existing worker's own token.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
kami
kami