F18. A single last_error slot destroyed causal evidence twice. Run 7
kept only the last of four failures. In run 11 a five-second retry loop
on a dead task pinned the slot for twenty-six minutes, so the live
task's own expiry was never visible at all, and run 12 lost diagnosis
time to the same thing before F58 removed the flood.
WorkerHealth now carries up to sixteen distinct observations, each with
its repeat count and first/last times. Collapsing is by message rather
than by position, because a loop interleaved with other failures would
otherwise still flush the ring. Eviction drops the least recently seen.
last_error and error_at keep their wire names and still report only the
newest failure, so nothing reading them has to change.
The ring lives in memory beside last_error and is not persisted, which
is the behaviour last_error already had across a restart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The supersession rule fires on TaskLeased, and a failed task is never
leased again. Run 12's rig task reached retry_limit still holding a
transaction whose commit the coordinator refuses permanently, so it kept
asking every five seconds with nothing that could ever change.
Terminal means terminal: TaskFailed now drops the transaction and
quarantines the session even when the anchor was pushed. Blocked keeps
the old rule, because a reopen still produces a successor that can pick
the anchor up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The late-handoff path fences on the epoch that expired. Once the task is
leased again under a different transaction, that epoch is two owners old
and the commit can never be accepted. The worker kept asking anyway,
every five seconds, holding the pane and pinning both ActiveTask and the
single last_error slot. Run 10's task did that for seven hours.
Proven live at 19:01:30Z: the lease expired while the anchor was
pushing, a successor took the task 13ms later, and the coordinator
refused the late commit with 409 lease not owned. The refusal is right.
The retry loop behind it was not.
TaskLeased now abandons a release transaction whose id the lease does
not carry, and quarantines its session. A successor pickup carries the
predecessor's own transaction id, so the recoverable predecessor F30
protects is left alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A release pushes the anchor first and commits second. When the lease
expired in between, the commit could never land: the worker sent the
epoch from w.leases, which the expiry replay had already deleted, and
the coordinator refused a /handoff without a live owned lease. Run 10
lost a finished task this way, its work sitting in the worktree until
retry_limit.
The epoch now belongs to the release transaction, so it survives the
lease. TaskReleased retains the ending epoch as Task.LastLeaseEpoch,
and lateHandoffAccepted lets exactly that owner commit while the task
is queued, unleased, and carrying no handoff of its own. A successor
that has already re-leased the task holds the lease, so the late
handoff loses. Store.Append's version fence settles the race, which is
why the late path skips the version check the worker cannot satisfy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
publishCaptures iterated every session the worker had, with no reference to
whether it still owned the lease. Run 10's task was blocked and unleased for
twenty-six minutes while this called the coordinator every five seconds and
logged "409 Conflict: lease not owned" each time.
The cost was not only the traffic. The single last_error slot stayed pinned to
that dead task, so run 11's own expiry reason was never visible, which is F18
turning a small leak into a blind spot.
A capture is lease-scoped. holdsLease names that invariant so the guard is
testable, rather than inline where a stub backend makes the test vacuous.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A refused handoff had no feedback loop. PrepareRelease read the report, the
parser refused it, the worker recorded the error in health, and the next
boundary read the same bytes and refused them again. Run 10 spent four leases
that way and the agent was never told anything.
The plan-progress path already had the answer: answerRefusedProgress says why,
drops the file, and lets the agent write a corrected one. The release path now
does the same, gated on a typed ErrInvalidHandoffAnswer so a transport or Git
failure keeps its retry.
This is the silent-loop shape CLAUDE.md names, in a path nobody had checked.
The three format fixes above it each removed one trigger; this removes the
loop.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
An implementer that finds the plan contradicted by the code had two options,
both bad: work around it silently, or improvise a different plan inside the
phase meant to execute one. PlanMismatch is the third.
The report carries an observation and nothing else. It may not propose a
replacement plan, because writing the next plan is the planning phase's work.
requested_action stays advisory: replan, research, or human_decision is a
recommendation, and Orchestra decides.
Staleness is checked before anything is recorded. A report names the plan ref
and the commit it was written against, both filled by the worker from what it
can verify rather than from what the agent asserted. A report against an older
plan says nothing about the current one, and one against an older tree may
already be fixed. Neither is replayed.
The reducer keeps two things apart that are easy to conflate:
mismatch recorded != plan superseded
A plan stops being accepted only when a replacement is actually sealed, so an
abandoned replan leaves the accepted plan and its verified progress intact. On
a real re-seal the old ref moves to PlanHistory and its progress stops counting,
while the verification events stay in the log as provenance.
human_decision never reopens. It blocks with a packet stating what was observed
and what it contradicts, and a human answer can resolve the contradiction
without resealing anything: the plan, its progress and the phase all survive,
and the answer outranks the plan where they differ. Turning every ambiguity
into a replan would put the planner above the person who set the goal.
The backward edge is Orchestra's alone. CanReopenPhase is separate from
CanTransitionPhase, which every path validating an agent's request uses, so
phase-request.json still refuses a move back. An agent asks by reporting a
mismatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A detailed plan that nothing enforces is a document. This makes the phases
executable: the implementer may write exactly one status, and every other
status is a conclusion Orchestra reaches by running the plan's own commands.
agent may request: ready_for_verification
agent may not assert: verified, awaiting_manual_verification, failed, skipped
The worker resolves commands from the coordinator, never from the request, so a
request cannot smuggle in a command the planner did not write. They run as argv
through exec with Dir set to the worktree, which is the quality gate's existing
envelope and not a weaker one. There is no shell, so a pipe is a literal
argument.
Project policy decides executable reach. registry.Project.Verification matches
argv positionally, and an absent policy refuses everything: a plan command is
agent-authored, so inheriting the operator-authored gate's reach by default
would be the wrong direction to fail in. A refused command is refused before
anything runs, and the refusal names the project and the command so the planner
learns its real reach.
Two bindings make the record mean something later. PlanRef, so progress earned
under plan A cannot survive into plan B. AtSHA, so "verified" does not outlive
the code that made it true: a record whose commit has moved is retained as
provenance and rendered as stale, never as a claim about the current tree.
Both are the same failure this codebase already fixed for reviews, which bind
to the commit they examined.
Manual steps hold a phase at awaiting_manual_verification. The sign-off is an
ordinary human decision whose subject carries the plan ref and the phase id, so
a later "looks good" on an unrelated thread cannot satisfy a gate nobody was
discussing.
A plan sealed before plan.md declares no executable unit, and says so: the
implement context states that phase progress is unavailable and the work
continues under the old semantics. Inventing phases it never had would be worse
than admitting it has none.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The plan artifact was Changes{Target,Intent} plus three string lists, every
entry capped at 500 single-line characters. That bound makes a specification
impossible: a phase cannot carry a code block, a paragraph of reasoning, or a
verification command with its own argument list. renderSealed then flattened
what little survived through collapse(), so an implement session received a
summary of a summary.
plan.md replaces it. Markdown, 128 KiB, no per-line cap, sealed through the
existing path under the existing PlanRef. The parser enforces the structure the
brief states: required sections, phases numbered from 1 with no gaps, Files,
Changes and Verification per phase, and at least one automated or manual check,
because a phase nobody can verify can never be established as done. Automated
entries are JSON argv arrays, so a pipe is a literal argument rather than an
operator. Headings inside fenced blocks are content, so a plan may show
markdown without parsing its own example.
Citations resolve at seal time against the accepted research, on the
coordinator, which is the only party holding ResearchRef. A plan resting on a
finding nobody recorded fails on the planner while its session is still alive
to be told.
The plan now renders byte for byte into the implement launch, and a rotated
successor receives the same complete document. That is the property the whole
change exists for. collapse() stays for research findings, which really are
short claims.
DecodeStoredPlan reads pre-markdown refs and renders them into the same type,
labelled, so nothing downstream branches on which era a plan came from. A
legacy plan carries no phases, which is honest: the old artifact never named an
executable unit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The brief at agentctx.go:167 advertised findings[].id and findings[].confidence
to every research session. The struct carried neither, so encoding/json dropped
both on every seal, silently, for as long as the schema has existed. A plan
phase had nothing stable to cite and no way to tell an observation from an
assumption.
Finding gains ID and Confidence. Ids are unique within an artifact and shaped
so "research:<id>" is unambiguous in plan prose. Confidence is fact, inference,
or assumption, matching the labels the output style already uses.
DecodeStoredResearch reads what is already in the CAS and backfills both.
Refusing an artifact sealed before this change would block every task whose
research predates it, including at rotation, where the agent that could fix it
is already gone. A backfilled finding is labelled inference rather than fact:
the old schema required evidence and made no verification claim, so upgrading
it on the way in would be the same class of lie this commit removes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The worktree survives a changes-requested round trip, so .orchestra/review.json
from the first review is still there when the second one starts. A reviewer
that writes .orchestra/done without rewriting it would have the earlier
findings sealed against the new commit, and submit binds whatever it reads to
the commit being submitted, so a stale pass is indistinguishable from a fresh
one.
Observed on the 2026-08-28 baseline run: the file from 10:55:45 was still
present when the second review session launched at 10:57:48. That reviewer did
rewrite it, so the run is sound, but nothing enforced it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The loop that reads merged pull requests was guarded by len(pullRequests) > 0
at a point 600 lines before the Gitea wiring that writes to that map. The
length was always zero, so the goroutine never started and a merged pull
request could never complete its task. Live on the first submission this
deployment made: PR #8 took a trusted comment and nothing moved.
Moving the block below the wiring fixes the ordering and avoids the race that
reading the map inside the tick would have introduced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Submission needs a sealed review, so a reviewing session that writes
.orchestra/done without .orchestra/review.json refuses every five seconds
with the failure recorded only in worker health. That is the silent loop this
codebase keeps producing: F39 at a phase boundary, F42 at the result commit,
and now the completion tail.
A review-artifact refusal is separated from every other submission failure,
delivered to the pane that can fix it, and clears the done marker so the
corrected file is what finishes the phase.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The completion tail ended at TaskCompleted with no pull request. Nothing in
the running system ever called the review or submission endpoints: the whole
event log holds zero ReviewRecorded and zero TaskSubmitted, so the merge
reflection, the publisher and the human trust boundary had no entry point.
Four links, in the order the tail needs them:
- finalize commits first and runs the quality gate against the committed
tree, so the gate result is bound to the commit being submitted.
CheckSubmission requires gate sha, review sha and head sha to be one
commit, which a gate run on the pre-commit tree can never satisfy.
- The worker seals the reviewer's findings and submits, through a new
/v1/federation/workers/<id>/submit. A blocking review returns the task to
implementation instead; a project with no forge still completes directly.
- The reviewing session is told where findings go. The brief asked for
findings and named no file, and it described a diff nobody supplied.
- GiteaPublisher.Push asks the forge what the branch holds before reaching
for a local checkout. A worker-owned worktree is on another machine and
has already pushed the commit; the coordinator has no such directory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F42, live on run 5's completion tail. The review agent wrote .orchestra/done,
the worker recognised it, confirmed the agent idle, and then failed the result
commit on every attempt:
stage result: The following paths are ignored by one of your .gitignore
files: .orchestra/done
.orchestra carries a .gitignore of "*" (internal/herdr/adapter.go:132), so the
marker is ignored, and git refuses an add whose pathspec names an ignored path.
The exclusion now names the directory. Reproduced against git 2.55.0 in a
scratch repo both ways, and the regression test uses the same constant the
worker passes to git.
The failure retried every five seconds for 22 minutes with the task stuck in
review and nothing observable outside the journal, because each identical error
overwrote the single last_error slot. That is F18, still open.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F41, live on run 5: the review agent produced nothing after 02:12 and the
02:46 renewal was granted anyway. The progress digest covered Claude Code's
status footer, and one of its fields ticked inside the window. Reproduced the
worker's stored progress_sha byte for byte from the live pane, so the branch
taken was progress != ProgressSHA, not IsBusy and not an empty baseline.
PaneProgress now cuts from the editor's lower rule and trims the spinner
summary and version notice above it. AgentStatus still reads the raw capture,
so the busy markers living in the footer are unaffected.
Lease TTL moves to 5 minutes, from domain.LeaseTTL, with renewal at half of
it. Reclaiming a stalled pane happens only at expiry, and 30 minutes per
window made run 5's stall unbounded in practice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Two halves of the same failure, live on run 5.
F38: the phase brief named .orchestra/research.json and described its contents
in prose, never its schema. The agent guessed dead_ends as strings where the
decoder wants {tried, why_failed} objects. The brief now carries the shape, and
a test decodes each documented shape with the same function the worker uses, so
a struct change that is not mirrored fails the build.
F39: the local artifact check refused the request through recordError alone.
answerRefusedPhase only ran on a coordinator 409, so a decode failure told the
agent nothing. The session sat at a boundary rewriting nothing, which is the
silent-loop shape the comment above that block warns about, reached by the one
path with no delivery. Both local refusals now reach the agent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Two gaps F33 left behind.
F34: the renewal gate exempts a lease with no baseline, so a pane that opened
and never started got a full free renewal period. That is the exact case the
gate exists to catch, and it happened live at 01:33:37: the stuck run 5 lease
renewed to 22:03 on a pane that had not moved since 01:13. Baseline the
progress hash at launch, where the pane is already being read.
F35: nothing could re-poke a live pane. Orchestra can put text in an editor and
be wrong about whether it landed, and the only recovery was to destroy the
lease and wait out expiry, roughly an hour. The new "resubmit" action presses
Enter on text Orchestra itself submitted. It changes no lifecycle state so it
appends no event, and it is fenced like an approval: a live worker-owned pane
at the capture revision the operator was looking at.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
A release transaction that never reached anchor_pushed has no artifact: tx.Ref
is empty and no successor can pick anything up. The event handler still kept
its session mapping alive on TaskReleased/TaskBlocked/TaskFailed, so once the
pane was gone the mapping was immortal. health() reports ActiveTask straight
out of w.sessions, so the coordinator saw the harness as permanently busy and
never leased to it again. It produced no log line at all.
Found live on workpc-claude, stuck at phase "prepared" behind a rejected
handoff artifact. Freeing it needed hand surgery on the worker's state file.
Keep the mapping only while an anchor actually exists. Drop the transaction
with it, since nothing can advance it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
F25. rotationTick returned early for claude before reaching federatedTurn,
which has one call site below that return. On the harness both burn-in runs
used, no phase request could ever be read and every human decision recorded
against a live session went undelivered. Claude still skips the occupancy
state machine below, because it owns its context rollover through the
installed hook. A turn boundary is not a rotation.
F26. The phase brief listed every domain-legal target, so run 4's frame
session read "research, implement" and asked for implement, which the
project's path refuses. The path is Orchestra's to know: the brief now names
one step and says a wrong target comes back with the right one.
F27. A refused request only reached recordError, leaving the agent to rewrite
the same rejected file forever with nothing telling it why. federation.
StatusError makes a 409 classifiable, and the refusal is delivered through
sendPrompt under the F20 guarantee. A transport failure is not an answer: the
request survives and the agent is told nothing.
The F25 regression test fails against the unfixed rotationTick.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
The brief told the agent to ask for a phase change and never carried the
asking. The agent asked in prose, no code represented the request, and the
session idled until its lease expired. That is what failed run 3.
F21. The agent asks with .orchestra/phase-request.json, and seals
research.json or plan.json where the phase it is leaving produces one. At a
verified turn boundary the worker checks the phase belief, the transition and
the artifact, then calls the coordinator with its lease epoch and a derived
operation id. AdvanceWorkPhase is unchanged, so a request cannot reach a move
the operator surface could not also make. Redelivery is idempotent.
F22. A session now records the phase it was launched to run. One that no
longer matches its task rotates with reason phase_changed, whether this worker
asked for the change or an operator made it.
F20. CLIAdapter.prompt sent handoff and rotation prompts without confirming
them, which is the failure F20 exists to catch. Fixed at the shared call site.
F23 needed no change. Issue comments already become decisions with no
submission, through Reconciler.Reconcile at PreLease and at every turn
boundary. The earlier finding searched internal/operations alone and was
wrong. Tests now cover the boundary it turns on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
F20. Only the launch confirmed its submit. A decision notice at a turn
boundary, and /clear or @HANDOFF.md during a context reset, were
fire-and-forget through the same transport that loses an Enter often enough
that the launch needed three resubmits. A lost Enter on the context-reset path
is the worst of them: it strands the session mid-rollover and nothing retries
it. LaunchConfirmer is therefore InputConfirmer, ConfirmLaunch is ConfirmInput,
and sendPrompt and sendLine both go through it.
Orchestra does not try to guarantee delivery of input it did not originate.
But it must never read that input as work, which is the F16 half. Burn-in run
3 stalled with an unexplained "go ahead and implement it" in the editor, and
the renewal check hashed the whole capture, so those keystrokes read as
progress and the lease kept renewing around an idle agent. PaneProgress drops
input lines from the capture, which the -J join makes exact: a wrapped input
block is one line beginning with the prompt marker.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
F16 shipped inert. renewLeases refuses to renew when the agent is idle and the
pane is unchanged since the hash recorded at the previous renewal, but
reconcileLeases rebuilt every lease from the coordinator's task list on each
tick and constructed a fresh struct, dropping ProgressSHA. The check therefore
took its no-baseline branch forever and renewed unconditionally.
Observed live: task 06G44JZB80MZBEY97196EZN8EC renewed at 09:31, 09:52 and
10:12 while its agent had been idle since 09:13, with a byte-identical pane
capture and no busy marker anywhere in it.
UsageBaseline and PickupAcknowledged were being wiped the same way, so a usage
receipt lost its baseline and a pickup could be re-acknowledged. The
coordinator is authoritative for the lease, not for what this worker observed
under it, so those three carry across a reconcile while the epoch is unchanged
and reset when it is not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A task that reached the router's MaxAttempts was permanently terminal.
TaskReleased only ever increments Attempt, TaskCorrected could not touch it,
and no HTTP route emitted a correction at all. The only way to work an
exhausted issue again was to invent a second task for it, which defeats
(source, external_id) dedupe and abandons the task's own history.
POST /v1/tasks/{id}/retry, full-control surfaces only. It requires the task
to be failed, unleased, and failed with reason retry_limit: restoring a retry
budget is not an answer to a failure that was not the budget running out. The
effect is one TaskCorrected naming that failure, setting state queued and
attempt 0 and clearing next_retry_at, failure_class and last_error. Task id,
source pair, goal, acceptance, decisions, work phase and artifact refs all
stay, and the original failure events stay in the log.
operation_id is required and makes the call idempotent, so a repeated request
cannot reset an attempt that has since started running.
This is RetryTask, not a generic correction endpoint: arbitrary task mutation
over HTTP is a different and much larger authority. It also does not address
F9, which is an operator releasing a lease someone else owns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
renewLeases renewed whenever a session existed and PaneCapture succeeded,
so a pane that opened and never accepted a prompt held its lease forever.
That is the mechanism behind the July stuck task: the launch failed and
nothing ever let go.
Renewal now needs the agent to be busy, or the pane capture to differ from
the one recorded at the previous renewal. The first renewal has no baseline,
so it records one and passes; the next must show movement.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Burn-in run 2 recorded TaskLaunchAcknowledged, opened a pane, and ran nothing
for fifteen minutes. The launch instruction sat in Claude Code's input editor
as "[Pasted text #1 +66 lines]" at zero tokens and zero elapsed. Two separate
bugs produced that.
The transport was wrong for the harness. TmuxBackend.Prompt writes the whole
instruction with send-keys -l and then sends Enter, and the TUI coalesces the
fast multi-line write into a paste that absorbs the following Enter. Launch
transport is now a backend property rather than one universal prompt format:
claude on tmux submits a single line pointing at .orchestra/launch.md, every
other harness keeps the inline path it was verified on. agentctx is unchanged
and the file still holds the exact bytes Orchestra rendered, so what the agent
receives is identical either way. Under the file transport a failed write is
now a failed launch, because there the file is the instruction.
The acknowledgement was also wrong. It meant "Prompt returned nil", not "the
harness accepted the prompt". Backends may now implement ConfirmLaunch, and
the tmux one polls until the input editor clears and the agent is observably
busy, blocked on approval, or at least no longer holding the text. An editor
that still holds the prompt at the deadline is a definite failure. The worker
kills the pane, drops the session so the retry starts clean, and returns
ErrPromptNotSubmitted, which classifies as prompt_not_submitted rather than
launch_uncertain. That class already falls through to TaskReleased, so the
existing retry path takes it and no lease is held on a launch that never
happened.
The confirmation bound is tunable because how fast a terminal harness reacts
is a property of the host. It is not a sleep before the submit: the submit is
deterministic, and this waits for the harness to visibly react to it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Burn-in run 2 ingested its task and then sat queued forever. Every herdr in
the live config declares quota_limit_5h and quota_limit_weekly, the event log
holds zero QuotaReported events, and QuotaSince reported an empty window as
unknown. QuotaAvailability fails closed on unknown, so no harness could ever
be leased, and the only producer of a receipt is a completed lease.
The event log is Orchestra's whole accounting source, so a window holding no
receipts is observable zero consumption. QuotaSince now reports known for an
empty window and for a harness that has never reported. A receipt that
declares its own consumption unknown still fails closed.
The refusal also lied about its cause. federatedAvailability collapsed a base
gate refusal into the federation health string, so router health said "stale
heartbeat or unhealthy local backend" while the heartbeat was one second old.
Availability gates now name themselves through an optional
ReasonedAvailability contract: quota refusals say whether usage is unknown or
the window is exhausted and by how much, and worker refusals distinguish an
unregistered worker, a never-probed backend, a stale heartbeat, a stale
health check, and an unreachable backend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A task in retry backoff was filtered out before the candidate loop, so it
recorded no rejection at all: queued, apparently assignable, and silent. That is
the exact shape that made F5 take a live session to diagnose. It now reports
"retry backoff until <time>".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
F7, security. An unset surface token makes the middleware skip its check, so a
full-control surface with no credential is an open control plane rather than a
closed one. With ORCHESTRA_TUI_TOKEN unset, any LAN caller could lease, release,
complete or block any task by declaring one header, which is how this session's
manual leases were issued. authz.RequireCredentials now refuses startup instead
of logging. Web is exempt: Sessions makes its login mandatory.
F5, lifecycle. router.go's silent `continue` was the first bug, not the
predicate behind it. Every eligibility gate now records a router.Rejection with
task, herdr and reason, exposed at GET /v1/router/health, reset per pass. No
gate was weakened: a direct Store.Lease succeeding proves the lease path, not
that eligibility should have selected that worker.
F8, correctness. Reconcile iterated every configured source for every task, so a
task's external id was looked up in whatever repository each source pointed at.
Once two repositories share an issue number, an unrelated human comment becomes
an authoritative decision for the wrong task. Reconciliation is now bound to
task.Source, the provider:project identity the ingest stamped, and a source that
cannot prove it owns the task is skipped. A task with no matching source
reconciles to nothing and still launches, because nothing to import is not a
failure to read.
The integration fixture ingested from "jsonl" while reconciling from "gitea",
which is exactly the shape F8 makes impossible; it now ingests from the source
it reconciles.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The burn-in pairs a coordinator with a worker, and step one of every run is
checking that both report the burn-in commit. That was only visible behind the
operator login, at /v1/admin/diagnostics and /v1/federation/workers, and the
worker never logged its own revision at all. Both now print it on startup, so
`docker logs orchestra-api` and `journalctl -u orchestra-worker` are enough.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v3 stack, previously an uncommitted working tree, plus this session's two
units and the burn-in instrument. This commit is the burn-in build identity:
coordinator and worker must both report this revision before a task is created.
Workflow (earlier sessions, uncommitted until now): human decision events and
reduction, source cursors and reconcile-before-launch, turn-boundary
reconciliation, internal/agentctx as the single renderer, ace-fca phases with
sealed artifacts, the trajectory gate, bounded grilling, independent review,
task pr enforcement, and human review reflection.
Capability restrictions at the agent boundary: an authz.Agent surface at
GatedWrite may ask and may not act. It also fixes two bugs the unit exposed --
gated surfaces could not reach the two endpoints written for them, and
RequestHumanDecision would block an unowned task while rejecting a question
from the session that did own it.
Turn-boundary reconcile-failure escalation: a streak of consecutive failures
asks the session to hand off, fenced on the lease epoch, with reconcile_failure
as a real handoff reason. The worker was dropping the coordinator's verdict on
the floor; it now acts on it.
Burn-in: herdr.WriteLaunchContext dumps the exact agentctx.Build result to
<worktree>/.orchestra/launch.md at every launch, local and federated. BURNIN.md
is the runbook. deploy/build.sh stamps both binaries from one commit.
go build, go vet and go test ./... pass, 20 packages.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Acts on the seven review comments on PR #1.
Design A is gone (comment 4). clients/ deleted rather than tracked: with
workers carrying cross-machine work the bridge is undeployed, which supersedes
the 2026-07-27 "keep through Phase 5" decision. CLAUDE.md, AGENTS.md and
AUDIT.md updated from "retained" to "deleted".
The harness-hook completion path is gone (comment 10). Investigation of the
live OpenCode QA run showed orchestra-worker owns completion end to end: it
watches for .orchestra/done, confirms via AgentStatus that the agent is not
busy, then posts through /v1/federation/* with both lease epoch and expected
version. The hook scripts used a different, older convention
(.orchestra-report.md) and posted to /v1/harness/complete, which had already
been reduced to a 410 stub - so that path could not have completed a task.
Nothing exercised it, because the live run never used it. Deleted: the three
deploy/hooks scripts, the 410 route, the unmounted harnessCompletion handler,
and its test. That test passed against a handler no mux routed to, which is
the exact "looks wired but isn't" pattern CLAUDE.md warns about; the
constant-time token compare added to it earlier today goes with it, having
never been reachable. /v1/harness/turn is untouched and still live.
Retired deployment files (comments 8, 12, 14): deploy/orchestra.service and
deploy/redeploy.sh (which sudo-installed to /usr/local/bin and restarted that
unit), plus deploy/docker-api-entrypoint.sh. The entrypoint was safe to remove
once its premise was checked: env vars reach the container through
`env_file: .env` in compose.yaml, not by sourcing /etc/orchestra/orchestra.env
- only config.jsonc is bind-mounted there - and Dockerfile.api's line 17
already sets ORCHESTRA_DATA/ORCHESTRA_PORT. Dockerfile.api now execs
/app/orchestra directly. orchestra-worker.service is a different, current unit
and is kept.
deploy/config.example.json deleted as a duplicate (comment 6); the annotated
.jsonc is the one registry.go points at, and its header no longer tells the
reader to copy the file that just went away.
Documentation corrected beyond the deletions:
- CLAUDE.md's deployment section claimed the container bind-mounts
/etc/orchestra:ro and its entrypoint sources the env file. Both wrong.
- AGENTS.md still described a systemd deployment on homesrv as of 2026-07-27.
- AUDIT.md's H5 row still described a "retained compatibility handler".
- deploy/DEPLOYMENT.md still named redeploy.sh as the deployment path.
- deploy/orchestra.env.example still cited EnvironmentFile=.
TOKEN_MINIMAL_WORKFLOW_PLAN.md (comment 2) is untouched: it and WEB_UI_PLAN.md
were both missed by REVIEW.md's documentation sweep, and reconciling a 534-line
forward-looking plan against AUDIT.md is its own task, not a review fixup.
Verified: go build ./..., go vet ./..., go test ./... all pass after the
deletions, and go list ./... has no node_modules entry. No live herdr or pane
was touched; nothing was deployed. The running image still predates this
commit until compose is rebuilt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GEugbHVYfAXFpTqDYbByEB
Acts on the 2026-07-30 senior review (REVIEW.md findings 1, 2, 4, 5, 7).
Docs (finding 1): CLAUDE.md and AGENTS.md both claimed Design B "has zero
clients - no worker binary exists". cmd/orchestra-worker/main.go is the
deployed worker, and the non-local-herdr guardrail has landed in
Coordinator.adapterFor. Both sections rewritten; AUDIT.md gains a matching
federation-status record. The Phase 5 retention / Phase 6 deletion decision
for Design A is preserved, not flattened.
clients/ un-ignored and tracked, including the .service unit and README:
deployed code belongs in version control. Design A is NOT deleted here.
progress.md (finding 2): the file was deleted after 636ed8a, yet CLAUDE.md
instructed every session to cross-check against it. References removed from
CLAUDE.md, AGENTS.md, internal/orchestrator/rotation_test.go (comment only)
and deploy/hooks/orchestra-codex-poll.sh; AUDIT.md now carries the log role.
web/go.mod (finding 4): a module stub ends the parent package graph at the
directory boundary, so go list ./... no longer yields
web/node_modules/flatted/golang/pkg/flatted. A build tag cannot work - the
package is in the package list before tags are evaluated. Local/CI-only
breakage: Dockerfile.api builds ./cmd/orchestra by explicit path and
.dockerignore already excluded node_modules.
orchestra-worker (finding 5): untracked (8.9MB, mode 100755, still on disk);
both binaries now gitignored.
Token compare (finding 7): cmd/orchestra/main.go:139,582 use
subtle.ConstantTimeCompare, matching the authz.go idiom. The token != ""
guard stays first, so an empty configured token still means auth-disabled
rather than auth-bypass. Three further plain != secret compares remain in
internal/federation/federation.go:343,346,368 - tracked, not fixed here.
Also included from the review pass: orchestrator.go records adapter-resolution
failures in SessionHealth.LastError instead of dropping them on a bare
continue, plus an Observed flag so lease-seeded health is not mistaken for a
live reading, with a covering test. GET /v1/tasks/<id>/health now returns a
record with last_error where it previously returned a bare 404.
REVIEW.md's own second pass claimed every checkable fact held up; four did
not. AUDIT.md never contained the false Design B claim (AGENTS.md was the
second copy), the guardrail is at orchestrator.go:312 not :309, the
progress.md site list missed the codex-poll hook, and only orchestra-worker
was tracked. Verified: go build, go vet, go test, and
go list ./... | grep node_modules all clean with every change applied
together. No live herdr or pane was touched; nothing was deployed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GEugbHVYfAXFpTqDYbByEB
All five defects filed while implementing B18, plus the router flake that
predated them. None of this has run on the deployed instance: the service
is stopped and /usr/local/bin/orchestra predates every change here.
B20 is the one that could silently defeat approvals. The capture revision
was UnixNano, so it changed on every read and said nothing about whether
the pane had changed; it is now an FNV-1a hash of the pane text, changing
iff the text does. The worse half was precedence: capture() preferred the
coordinator over a published worker capture, handing Queue a timestamp the
owning worker's staleness check could never match, so every federated
approval resolved "stale" and the keystroke never happened. Worker captures
now win — their existence means a registered worker owns that pane — and
capturePane follows the same precedence via Capture.Source rather than
guessing.
B19 was filed as "federated approvals emit no event", which overstated it:
the resolution half already existed, and correctly fires only on an
acknowledged worker report. The missing half was the request. Server.action
now appends ApprovalRequested at queue time, subject_ref set to the command
ID the later resolution carries. If that append fails the queued command is
resolved "rejected" — a keystroke that left no audit trail must not run.
B21 bounds the command list: resolved commands prune after 30 minutes on
both Queue and Commands, pending ones never at any age, since dropping one
would discard an operator decision. The persistence half stays open and is
recorded as such — captures and commands are still in-memory only.
S12 splits ORCHESTRA_NTFY_TOKEN, which was both the secret handed to the
ntfy server and a valid inbound credential for the ntfy surface; the latter
is now ORCHESTRA_NTFY_SURFACE_TOKEN. Breaking: a deployment relying on the
old dual use has no inbound gate until it sets the new variable. S13
deletes the dead auth() copy of the authorization policy.
The router flake was in the test, not in assignment. Store.Tasks() ranges a
map, and the assertion indexed two separate Tasks() calls, failing whenever
the orderings disagreed; instrumenting it showed a valid TaskLeased and a
genuinely leased task on every "failing" run. It now snapshots once and
asserts that exactly one task is leased, and passes at -count=60.
AUDIT.md records what is still not done: the deployed env and binary, the
live re-verification B13-B17 has always lacked, and two operational faults
found in the journal that block it — all six herdrs are refusing
connections, and ntfy delivery is failing 403 on every send.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
The UI is a full control plane: it can create tasks, release or complete
them, and inject approval keystrokes into live panes. authz.HTTP did cover
it (an absent surface header defaults to Web), but the gate is opt-in and
the deployed env sets no tokens while binding all interfaces, so in
practice it was reachable unauthenticated from the LAN. Setting the token
alone did not work either: a browser cannot put a bearer token on a
document load, so the UI would 401 on index.html.
- ORCHESTRA_WEB_TOKEN is now mandatory; startup fails rather than silently
serving an open control plane.
- authz.Sessions issues random values stored SHA-256-hashed, with a TTL,
so a leaked snapshot yields nothing usable.
- POST /v1/ui/session verifies the token in constant time and returns it
as an HttpOnly, SameSite=Strict, Secure cookie. This is a presentable
form of the same credential, not a new authority.
- HTTPWithSessions accepts that cookie in place of the bearer token, and
only for the Web surface. The login endpoint and non-/v1/ GETs (the SPA
shell) are exempt by necessity; every /v1/ control path stays gated.
Note this is a breaking config change: .orchestra-config/orchestra.env
sets no tokens, so the service will not start until it does, and setting a
Web token newly gates the other /v1/ surfaces that default to Web.
AUDIT.md is reconciled against the code rather than against itself. B14,
B15 and B16 are closed with their evidence; B17 is closed on the worker
path only; the stale claim that B13 was open is corrected. Adds the
previously undocumented command channel and web UI, and files what that
implementation pass surfaced: federated approvals emit no event (B19), the
local capture revision is a timestamp rather than a change counter and can
silently defeat approvals (B20), the command queue never prunes (B21), the
ntfy token serves two unrelated purposes (S12), and a dead copy of the
authorization policy sits in main.go (S13).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
Introduces the browser-facing surface and the worker-side protocol that
backs it:
- internal/ui: joined read model plus per-task lifecycle and approval
controls, kept separate from the raw endpoints workers and harnesses
depend on.
- internal/webui + web/: Vite/React app, build output embedded via
go:embed and served as an SPA fallback.
- federation: per-(worker, task) captures with a monotonic revision that
advances only when pane text actually changes, and a command queue
restricted to grant_approval / deny_approval, each bound to the capture
revision the operator acted on.
- orchestra-worker: publishes captures and executes commands only after
re-reading the pane and confirming the revision still matches. Sends
keystrokes only for a visible y/n prompt or OpenCode's fully labelled
selector, and refuses to deny through that selector rather than guess
at unobservable navigation.
This is the ownership boundary AUDIT.md's B14 and B17 call for: approval
becomes an explicit, revision-bound operation executed by the worker that
owns the pane, instead of a side effect of prompting over a
coordinator-driven remote socket.
Also ignores the web build inputs and outputs. node_modules ships vendored
Go packages, so go build and go test walk into it if it is merely
untracked; both node_modules and .node_modules are excluded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
Coordinator.Soft (default 0.55, ORCHESTRA_OCCUPANCY_SOFT) makes rotate()
and TurnDecision request a handoff advisory-only once occupancy crosses
the soft threshold, well before Hard forces a release/rotation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Register() previously trusted a self-declared id and self-chosen token
from any caller, and let a second caller silently hijack an existing
worker id by re-registering it with a different token. Adds an optional
pre-shared AdmitToken (ORCHESTRA_FEDERATION_ADMIT_TOKEN) and requires a
same-id re-registration to present the existing worker's own token.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
The 1s ticker's own ExpireLeases call almost always won the race against
the coordinator's 30s Monitor loop, so the coordinator's ExpireLeases saw
nothing left to expire and never reached its session-kill path — silently
orphaning herdr panes past their lease TTL whenever a coordinator is
configured. AUDIT.md called this "harmless" on the assumption CAS merely
picked a winner; the real effect is the pane-kill side effect never firing.
Now the ticker defers reclaim entirely to the coordinator when one exists,
keeping only AssignPending as a periodic retry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Adds an optional "harness" field so codex/opencode completions route to
their own Usage readers instead of always assuming Claude's transcript
format; unblocks the server side named as open in progress.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Brief.Git was a single GitSync read from ORCHESTRA_DATA (never a git
checkout), and completions were counted but discarded their report_ref/
receipt. Brief.Git is now keyed by project ID and built from each
project's real repo; GitSync gained Ahead/Behind vs upstream; Brief now
carries Receipts pulled from each TaskCompleted payload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Closes S4 (AUDIT.md): Fanout.Run returned on the first sender error,
permanently ending notifications for the process lifetime after one ntfy
hiccup. Failed sends now go through an OnError hook and the loop
continues. Also persists the delivery cursor to a file next to
ORCHESTRA_DATA so a restart resumes from the last delivered event instead
of re-notifying the entire log from seq 0. Adds the package's first test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
kami
kami