Neither will be manufactured: a second task hitting the same failure
exercises the eligibility transition, and a genuine repair exercises manual
intervention provenance.
Also records why the reasons check earned its place. An inference drawn
from aggregate numbers was wrong, and the pure explanation contradicted it
with counted facts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Reasons are the auditable explanation of a pure decision, so one that
describes the architecture rots the moment the architecture moves. The
operational refusal asserted that manual interventions were recorded by no
event type: true when written, false the day OperatorInterventionRecorded
landed, and still printed under every refusal after that.
It now reports the counts and the thresholds they missed. Why a count is
zero is not this function's business, since no intervention happening, none
being recorded, and none being migrated all read the same from here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
An intervention is an action required to recover, repair, unblock or correct
behaviour that should have proceeded on its own. A deploy restart, a planned
upgrade and burn-in setup are none of those, and recording them would inflate
operator cost with work Orchestra is expected to need.
Policy rather than schema. worker_restart is the one kind that spans both
sides, and it stays one kind until someone actually misuses it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A projection must never manufacture provenance to make evidence easier to
classify. task=None is better than a confident lie.
Also records that signatures are frozen at write time, so an old event never
changes meaning because normalization code changed. Repairing historical
signatures is an explicit migration, not a different projection result from
the same log.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Both defects came from the first real run of slice B against live data, and
neither was visible in a unit test written from the design.
The ring is a bounded history, not a set of live conditions. A quiet timeout
closed an incident, its entry stayed in the ring because nothing evicts it,
and every later heartbeat opened the same incident again: three signatures,
four incidents each, from failures that never happened twice. An incident
now opens only when the entry actually advances past what was already
accounted, and the high-water mark survives the close.
The ring also outlives the work it describes, so attributing its entries to
whatever the worker is running now invented an association. The task is read
out of the message, and only a failure that names no task belongs to the
current lease. An incident that names an older task has no live lease to
bound it, so it closes on quiet timeout rather than on the next epoch change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The ledger's first live run showed it: a pane name carries the task id in
lower case, so "phase rotation ...: pane orchestra-<task>-<sha>:1.0 still
holds input" signed differently on every task and could never accumulate
recurrence across them. The id pattern is case-insensitive now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Slice B, second half. OperatorInterventionRecorded is the one command for
saying "I fixed this by hand": a manual repair happens outside Orchestra by
definition, so the only honest way to have the evidence is for the person
who made it to state it. Inferring "an operator probably intervened" from a
gap would put guesses into the record the ledger is built from.
The debt projection now consumes both new kinds. A closed incident is one
observation carrying its repeat count as intensity, so recurrence stays a
count of independent incidents: 301 repeats on one lease and 2 on another
is a recurrence of two with an intensity of 303, not a recurrence of 303.
Both kinds were previously reported as holes in the system. They are
ordinary evidence now, so their absence from a history is a fact about that
history, and the gap list says so.
The worker also stamps a per-process incarnation on registration and every
heartbeat. Nothing else on the wire distinguishes a restarted worker from a
running one, and an incident cannot outlive the process that reported it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Slice B, first half. The F18 ring is bounded, lossy and local, so the debt
ledger reported it as a gap about itself. Two events make it durable:
ObservationIncidentOpened at first sight, appended immediately so a
coordinator that dies mid-incident still leaves the fact that it existed,
and ObservationIncidentClosed carrying the aggregate.
The rules are what matter. Repeats update the aggregate and append nothing,
because run 11's 409 loop was one incident with an intensity of 301 rather
than 301 pieces of evidence. Absence from the ring closes nothing, since a
bounded history evicts as easily as it recovers. An incident is scoped to
its lease and closes on lease end, epoch change, worker restart, or, for
observations with no lease to bound them, on last_seen going stale. A ring
entry that is evicted and recreated accumulates: 34 then 3 is 37.
Also collapses three drifted copies of the list of events that carry no
task into one predicate. Adding a type to two of them left it rejected by
the third, which is how the first version of this failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Run 20 showed the leak precisely: the reopen rotated the session, and 31
seconds later the planning session that replaced it recorded phase-2 of the
plan being replaced. The implementer's .orchestra/plan-progress.json had
outlived it in the worktree, and the successor executed it.
Two changes. Phase verification now belongs to the implement phase and is
refused anywhere else, which covers every worker and every path rather than
the one that produced it. The rotation also drops the ended session's
request files, so a successor never inherits work from a trajectory
Orchestra has already invalidated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The first fix wrote the projection and lost it at the rotation the reopen
causes, so the planner still got nothing. Both rounds passed their unit
tests; what separated them was where the assertion sat in the sequence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The clearing rule sat below the reducer switch, so it ran for every event
rather than for the correction it was written for. TaskReleased found the
task unblocked and erased the contradiction, which is the rotation the
reopen itself causes: the planning session launched one lease later and was
told nothing again.
Scoped to TaskCorrected, and the store test walks the real sequence
(mismatch, reopen, release, lease) rather than reading the projection at
the moment it is written.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F66, found live on run 20. A replan reopens the plan phase and the session
it convenes was given the ordinary plan brief and an empty template: no
mention that a plan already exists, what contradicted it, which phase, what
evidence, or that its output supersedes an accepted plan. All of it was
already durable on PlanMismatchRecorded and none of it reached the agent,
so nothing stopped the replacement from sealing with the same
contradiction in it.
The contradiction is now projected onto the task and rendered above the
sealed artifacts, because it changes how they should be read. Its lifetime
is bounded at both ends: a sealed replacement answers it, and so does a
human reply to the stop it caused. Only the phases a mismatch can reopen
render it, since an implementer looks at the code rather than at an account
of the last contradiction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F67. TaskCorrected cleared the standing question when a task resumed and
left Blocker and BlockReason in place, so task 29 ran through implement,
review and submission still reporting block_reason plan_mismatch. Every
surface that reads the projection rather than the event log showed a stop
that had already been answered.
The blocker has the same lifetime as the question beside it, and is now
cleared with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Both halves proven live. A human_decision mismatch stops the task, the
blocker packet states the contradiction, and a Gitea reply resumes the same
plan in 55 seconds with no reseal. A replan reopens into plan while the old
plan stays accepted with its progress, and sealing the replacement moves
plan_ref, retains the old one in history, stops the old progress counting,
and hands the successor the replacement whole.
F66 recorded: the planning session a replan convenes is told nothing about
the contradiction it exists to fix, though all of it is durable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F65, found live on run 20. A plan mismatch asking for a human decision
recorded its observation, then failed to block the task: Store.Append fences
every lifecycle event on a leased task against the current owner and epoch,
and this TaskBlocked carried neither. The task kept implementing while the
contradiction sat durable in the log, and the agent was told its report was
refused. The trajectory gate had the same omission.
The human-decision path already did this correctly and explained why in a
comment. That comment is now a helper all three call.
The tests could not have caught it. planWith never leased its task, so
every plan test ran in a state no agent can be in, which is exactly what
the lease helper's own comment warns against. It leases now, and the
mismatch block test fails without the fence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The ethos console sat undeployed for hours while two coordinator deploys
went out, because both used --no-deps orchestra-api and the web image is
built separately. Checking the commit does not catch it; checking the
served bundle does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F64. A plan mismatch that asks for a human decision blocks the task, and
nothing came back. Two independent gaps, either one enough to strand it:
the reconciler ran only before a lease and at a turn boundary, so a blocked
task's reply was never even read, and ResumeAnsweredBlockers listed two
block reasons, not this one. PlanMismatchAnswered had no caller anywhere.
Blocked tasks awaiting a reply are now reconciled on their own loop, the
same reasoning the submitted-work loop above it already uses: a task that
cannot be leased cannot be reconciled behind a pre-lease hook. One
predicate, BlockReasonAwaitsReply, now names the set for both loops so they
cannot drift apart again.
The existing test asserted the predicate and never the resume, which is how
this survived. It asserts the resume now, and fails without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F63, found live on run 19. A manual check on these projects is a human
reading what the code prints. RecordPlanPhaseVerification asked only
whether a sign-off for that plan and phase existed, and one exists forever,
so rerunning a phase's automated checks at a new commit carried the human
half along with it. The rig proved it twice: two operator commits and two
re-verification requests, each coming back verified without anyone looking.
The reducer now records which tree the human confirmed, the record carries
it forward as provenance, and a run whose commit does not match it waits
for the human again. A sign-off given before any run has no confirmed tree
and still counts, so the ordinary ordering is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The keyed seam holds. Automated checks alone leave a phase awaiting, a
generic approving comment satisfies nothing, and the keyed form verifies
exactly the phase it names under the plan ref Orchestra supplies. A
re-verification request reruns the plan's own commands at the new tree and
writes a record bound to it.
F63 is what the rig found. A manual sign-off is consulted by existence
alone, so it survives every later tree the human never saw, while the
automated half is re-established at each new SHA. The proposed fix stamps
the satisfied record's sha on the decision and requires it to match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The rung the last two runs missed. A five-phase task with ten named checks
kept the implement phase open long enough to rotate inside it. The successor
picked up in implement and its launch context carried the whole sealed plan,
phase-1 and phase-2 as verified, phase-3 as the first unfinished phase, and
the current human authority.
Unasked-for bonus: the progress block renders SHA staleness itself, naming
the tree each phase was verified against and the tree it is now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The rig was not needed. An implement to review phase change asked for the
handoff through the ordinary path, and the agent ignored it, which is the
exact shape F62 was written for. Every assertion held: a causal release
rather than an idle expiry, four renewals during the bounded wait, one
resend carrying the original reason, the timeout class at 10m5s, no stale
transaction, and a successor that leased normally a minute later.
Also recorded: a five-phase plan does not lengthen the implement phase, and
the state-file lever cannot be driven with systemctl restart alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F62. The rotation is agent-driven: the worker asks, and the agent must write
its handoff. When the agent never does, renewals stopped on the ordinary
progress gate, the lease expired, and the task lost an attempt with nothing on
record saying a handoff had ever been requested. Run 16 showed only "agent
status idle and pane unchanged", 34 times.
The request is now stamped, and the wait around it is bounded. While Orchestra
is explicitly waiting the lease renews, because a quiet pane is the answer the
agent was told to give. The request is re-sent once after four minutes, with
the reason it was first asked with. At ten minutes the worker nacks with
failure class handoff_unanswered, and the coordinator releases the task naming
that cause instead of letting the lease die as generic idleness.
The class is known to DebtClassForFailureClass, so a harness that ignores
handoff requests accumulates as its own debt item rather than hiding inside
lease_expired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F57 to F60 close the release-transaction family, F18 closes the single
last_error slot, the debt ledger exists as a read-only projection, and
the operator console is rebuilt on ethos.
The part worth acting on is what the UI could not do honestly: no
web-facing human-decision write path, no keystroke forwarding, context
occupancy trapped in herdr, project configuration unserved, and no
federated handoff request. F61 and F62 are recorded and unbuilt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A successor inherits the whole sealed plan. Phase progress is withheld
from a review successor on purpose, so the rung still needs an
implement-phase successor. F62: a requested handoff nobody answers is
invisible, and its only consequence is an expiry indistinguishable from
an idle one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Nine screens against orchestra-ui-spec.md and the accepted mockups, on
the ethos design system: signal violet #8F7AE5, the routing fork motif,
the 64px rail and 56px top bar, mono for every machine value and sans
for every human one.
Each screen was built by its own agent against a fixed foundation, so
the shell, tokens and primitives have one author and the screens cannot
drift into nine dialects.
Real data only. Where no endpoint exists the screen says which one it
needs instead of inventing a value. That is most of what was learned
here: Steer / Correct is disabled because nothing records a human
decision from the web, take-control is disabled because nothing
forwards keystrokes to a pane, context occupancy is missing from three
screens, and projects can show no repo, remote, quality gate or
verification policy because those live only in config.jsonc.
Three bugs the render caught that no computed value would have:
The previous stylesheet fought every shared class name and leaked
properties the new rules never mention, which is how position:fixed
survived on .topbar. It is now scoped under .legacy and applies only to
the login route, which also stops its green accent and its
backdrop-filter from reaching the console.
Go marshals a zero time.Time as "0001-01-01T00:00:00Z" and omitempty
does not omit a struct, so absent timestamps arrived populated-looking
and rendered as "739855d ago". Stripped once at the API boundary every
screen reads through, with a test.
Long machine ids overflowed their cards and painted under the next one.
Verified by rendering: chromium screenshots of the dashboard, tasks,
task detail, terminal, workers and review at 1440px, and the dashboard
at 390px. Geist is still not on disk, so both stacks fall back to the
system faces.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Not my work. These nine files sat uncommitted in the shared checkout
while another session worked on them, and the UI redesign that follows
rewrites web/src/main.tsx and web/src/style.css. Committing first means
that work is recoverable rather than overwritten.
Contents, by inspection rather than by authorship: whitespace
normalisation and edits across main.tsx, 568 added lines of style.css,
client and client test changes, the orchestra-user line in build.sh, and
docs updates to AGENTS.md, AUDIT.md, DEPLOYMENT.md and the env example.
Committed at the operator's explicit instruction.
Slice one of DEBT-DESIGN.md, with the four amendments applied. It writes
nothing: no new event types, no scheduling, no clustering, no
maintenance tasks. The point is to find out whether the model can
represent debt this project already knows about, before committing to a
durable schema.
The projected type lives in domain and the fold lives in store, so the
first implementation does not bake a read model into the command layer.
operations owns the one action that exists, CheckDebtEligibility, which
is a pure function returning explicit reasons like CheckSubmission.
Signatures carry their version in the string. Normalization rules will
change, and without a version that silently regroups history and moves
the recurrence counts eligibility was already decided on.
Observations require exactly one of event_id and legacy_ref. Imported
Fxx history predates the events that would justify it, and a fabricated
event id would break the provenance rule the ledger exists to enforce.
Incompleteness is reported, not hidden. Manual interventions and worker
observations are carried by no event type, so the ledger names both as
non-durable gaps rather than reading as "no operational cost". The
operational refusal reason says the intervention count is structurally
zero on every current log.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Nine questions answered against the tree at 3c7cf95, no implementation.
The ledger is two event types, one projection beside Task, one pure
eligibility function and one read-only endpoint. Nothing in the task
lifecycle changes.
The first slice writes nothing: a projection over the existing log that
must reproduce the release loop, the opencode adapter gap and the
retry-idleness dynamic from runs 10 to 14. A model that cannot represent
debt already known is wrong before any schema is committed to.
Also corrects run 14: the OpenCode Zen free tier is not blocked, the
selected model is.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F60, and the general rule F58 and F59 were reaching for one case at a
time: a transaction must settle or be abandoned deterministically, and
must never spin on an answer that cannot change.
Terminal now means failed or completed. Both drop the transaction and
free the session; nothing will ever lease either task again.
Blocked keeps the transaction, because a reopen returns the task to the
queue and that exact owner can still commit. TaskBlocked therefore
retains the ending epoch the way TaskReleased already did, or the
late-handoff path would have nothing to fence against after the reopen.
A refusal parks the commit instead of retrying every five seconds. It
is the coordinator's answer about who owns the task, so it stays true
until an event about that task arrives, and any such event un-parks it.
A reopen arrives as TaskCorrected, so the rule cannot be a list of
event types. Backoff runs 30s to a 5 minute cap.
A transport failure is not an answer and keeps retrying at once. That
distinction is the whole reason the park keys on a 4xx StatusError
rather than on any error at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F18. A single last_error slot destroyed causal evidence twice. Run 7
kept only the last of four failures. In run 11 a five-second retry loop
on a dead task pinned the slot for twenty-six minutes, so the live
task's own expiry was never visible at all, and run 12 lost diagnosis
time to the same thing before F58 removed the flood.
WorkerHealth now carries up to sixteen distinct observations, each with
its repeat count and first/last times. Collapsing is by message rather
than by position, because a loop interleaved with other failures would
otherwise still flush the ring. Eviction drops the least recently seen.
last_error and error_at keep their wire names and still report only the
newest failure, so nothing reading them has to change.
The ring lives in memory beside last_error and is not persisted, which
is the behaviour last_error already had across a restart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The supersession rule fires on TaskLeased, and a failed task is never
leased again. Run 12's rig task reached retry_limit still holding a
transaction whose commit the coordinator refuses permanently, so it kept
asking every five seconds with nothing that could ever change.
Terminal means terminal: TaskFailed now drops the transaction and
quarantines the session even when the anchor was pushed. Blocked keeps
the old rule, because a reopen still produces a successor that can pick
the anchor up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The late-handoff path fences on the epoch that expired. Once the task is
leased again under a different transaction, that epoch is two owners old
and the commit can never be accepted. The worker kept asking anyway,
every five seconds, holding the pane and pinning both ActiveTask and the
single last_error slot. Run 10's task did that for seven hours.
Proven live at 19:01:30Z: the lease expired while the anchor was
pushing, a successor took the task 13ms later, and the coordinator
refused the late commit with 409 lease not owned. The refusal is right.
The retry loop behind it was not.
TaskLeased now abandons a release transaction whose id the lease does
not carry, and quarantines its session. A successor pickup carries the
predecessor's own transaction id, so the recoverable predecessor F30
protects is left alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A release pushes the anchor first and commits second. When the lease
expired in between, the commit could never land: the worker sent the
epoch from w.leases, which the expiry replay had already deleted, and
the coordinator refused a /handoff without a live owned lease. Run 10
lost a finished task this way, its work sitting in the worktree until
retry_limit.
The epoch now belongs to the release transaction, so it survives the
lease. TaskReleased retains the ending epoch as Task.LastLeaseEpoch,
and lateHandoffAccepted lets exactly that owner commit while the task
is queued, unleased, and carrying no handoff of its own. A successor
that has already re-leased the task holds the lease, so the late
handoff loses. Store.Append's version fence settles the race, which is
why the late path skips the version check the worker cannot satisfy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Thirteen rungs, ten defects found and fixed, both halves deployed at
orchestra-plan-v1. The handoff records what the defects had in common, why the
mismatch rung took four attempts, and the two open items that should be fixed
before anything runs unattended.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Ten defects now, all fixed. The tenth was a five-second retry loop against a
blocked task that ran for twenty-six minutes and pinned the single last_error
slot to a dead task, which is F18 turning a small leak into a blind spot.
Run 11 went on through review, submission and merge without intervention, so
the frozen lifecycle is re-proven above every fix in this session.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
publishCaptures iterated every session the worker had, with no reference to
whether it still owned the lease. Run 10's task was blocked and unleased for
twenty-six minutes while this called the coordinator every five seconds and
logged "409 Conflict: lease not owned" each time.
The cost was not only the traffic. The single last_error slot stayed pinned to
that dead task, so run 11's own expiry reason was never visible, which is F18
turning a small leak into a blind spot.
A capture is lease-scoped. holdsLease names that invariant so the guard is
testable, rather than inline where a stub backend makes the test vacuous.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The last rung. A plan sealed through the phase endpoint asserted a helper the
script does not define, and the implementer reported it rather than improvising
or working around it.
Every authority property held. The report carried an observation and no
replacement plan. Orchestra chose the transition. The old plan stayed accepted
until the replacement sealed, then moved to PlanHistory, and progress did not
carry across the ref change. The replacement rendered verbatim into a fresh
implement session.
The three failed attempts are kept, because each was the agent being right: a
file it can restore is reversible, a human correction is authoritative, and a
competent planner does not author a false premise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Twelve rungs proven, one not. The typed mismatch never fired, and both
attempts to force it tested the wrong thing: a file the agent can restore is
reversible, and a human correction is authoritative. Neither is a case where
the repository contradicts the plan.
Nine defects, all fixed. Three of them are one shape, a constraint the code
enforces and no brief states. The fourth is what made them expensive: a
refused handoff had no feedback loop, so the same rejected bytes were re-read
at every boundary until the task hit retry_limit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A refused handoff had no feedback loop. PrepareRelease read the report, the
parser refused it, the worker recorded the error in health, and the next
boundary read the same bytes and refused them again. Run 10 spent four leases
that way and the agent was never told anything.
The plan-progress path already had the answer: answerRefusedProgress says why,
drops the file, and lets the agent write a corrected one. The release path now
does the same, gated on a typed ErrInvalidHandoffAnswer so a transport or Git
failure keeps its retry.
This is the silent-loop shape CLAUDE.md names, in a path nobody had checked.
The three format fixes above it each removed one trigger; this removes the
loop.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The prompt gave "at most 200 characters" for NEXT, WHY and REMAINING. The
validator applies it to OPEN Q and LEARNED as well, and run 10 lost a lease to
a 219-character OPEN Q against a limit nobody had stated for that field.
Third instance of one shape: a constraint the code enforces and no brief
mentions. The other two were the research finding id and the dead-end
separator.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
f4dbcf7 taught the verified branch about staleness and left the pending one.
Run 10 showed a successor "automated checks passed at 94bd45c3b5d6" while HEAD
was 7d04aef: the automated half of that phase was established against code
that had since changed, and nothing said so.
A human about to confirm the manual steps of a phase deserves to know the
automated half no longer describes the tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
98f1b2d fixed the arrow and left the other half. "tried" and "failed because"
restate the field names either side of a separator that already says what the
line means, and the parser required both.
Run 9 lost three leases to this. Every refused line carried exactly the data
the fields want:
tried run line ["bash", "scripts/test_healthcheck.sh"] for phase-3
-> refused, policy allows only bash -n on one file
tried finding ids "F1".."F8" in research.json
-> schema requires lowercase letters, digits, dash or underscore
Both prefixes are now optional. A line with no cause and effect is still
refused, because the split is the thing being validated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The brief tells the planner "a command outside its policy is refused when you
seal, not later". It was not. The only caller of VerificationPolicy.Allows was
PlanPhaseCommands, which runs when the implementer asks to verify: one phase,
one session and one rotation after the planner could have fixed it.
Run 9 sealed ["bash", "scripts/test_healthcheck.sh"] against a policy that
allows neither shape, and the phase request was accepted.
The check now runs beside citation resolution, on the coordinator, where the
project is already in scope. A project with no verification policy can still
seal a plan; it cannot seal one that declares run: lines, which matches what
an absent policy already meant at verification time.
Test fixtures gained a policy for the same reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Run 9's planner wrote a dead end as "tried X -> failed because Y". The content
was exactly what the protocol asks for. The parser split on "→" only, refused
the handoff, and a refused handoff fails the release rather than the turn, so
the lease expired with the work intact and unrecorded.
Accept either arrow. A separator is not the thing being validated.
The prefix stripping was also wrong on the right-hand side: it trimmed
"failed because " from an untrimmed string that started with a space, so every
parsed dead end kept the literal prefix in why_failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Run 9's research seal was refused for writing "F1" as a finding id. The rule
is real and the message was precise, but no brief had ever stated it: the
shape block shows keys and types, and a format constraint is neither.
The agent recovered in fifteen seconds, so this cost one boundary rather than
a run. It is still a refusal nobody had to earn, and the same shape hid F38 a
few runs ago.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Run 8 rendered three phases as "verified at 14654d6cab63, stale because the
tree is now at 56d9b9acdd59" and then printed "Every phase is verified" under
them. The staleness was honest and the conclusion was not.
Stale still counts as done. The usual reason a phase goes stale is the next
phase's own commit, and treating that as unverified would send the implementer
back to phase 1 after every commit. What the successor needed was the names:
which phases were verified against code that has since changed, and an
instruction to reverify what its work touches.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The gate had two live consumers and no producer. store.go resolves a pending
sign-off by subject, and manuallySignedOff checks for a prior one, but every
comment Orchestra imported was hardcoded to operator_instruction. A phase
carrying a manual step could reach awaiting_manual_verification and never
leave it.
A comment whose first line reads "orchestra verify <phase-id>" now approves
that phase. Orchestra supplies the plan ref from the task's own accepted plan,
so the approval binds to the plan that was current when the human wrote it.
Everything else still lands under operator_instruction, which is what keeps a
generic "looks good" from satisfying a gate nobody was discussing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Delivery merges the task branch, and that branch carries the "orchestra:
TASK.md" commit. Master therefore ends up holding the previous task's TASK.md,
and the next worktree branches from it.
writeTaskFile returned early on os.Stat, so it left that inherited file in
place. The worker then hashed the current task and every immutability check
failed with "TASK.md changed" against a hash for a task nobody was running.
Releases failed, rotation never relaunched, phase requests were never read, and
the task died on retry_limit without leaving research.
Presence is not identity. Compare content, and rewrite when it differs.
Found live in run 7, the first task to start after run 6's pull request merged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The tag orchestra-conformance-v1 marks the full original lifecycle, live-proven
by run 6. Both deployed halves sit there deliberately.
The four plan-machinery commits above it pass build, vet and test, and have
never run against a live agent. The handoff records what they need before they
can: a verification policy on test-e2e, both halves rebuilt together, and F18
fixed first, since the burn-in will generate the retry loops it exists to make
visible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
An implementer that finds the plan contradicted by the code had two options,
both bad: work around it silently, or improvise a different plan inside the
phase meant to execute one. PlanMismatch is the third.
The report carries an observation and nothing else. It may not propose a
replacement plan, because writing the next plan is the planning phase's work.
requested_action stays advisory: replan, research, or human_decision is a
recommendation, and Orchestra decides.
Staleness is checked before anything is recorded. A report names the plan ref
and the commit it was written against, both filled by the worker from what it
can verify rather than from what the agent asserted. A report against an older
plan says nothing about the current one, and one against an older tree may
already be fixed. Neither is replayed.
The reducer keeps two things apart that are easy to conflate:
mismatch recorded != plan superseded
A plan stops being accepted only when a replacement is actually sealed, so an
abandoned replan leaves the accepted plan and its verified progress intact. On
a real re-seal the old ref moves to PlanHistory and its progress stops counting,
while the verification events stay in the log as provenance.
human_decision never reopens. It blocks with a packet stating what was observed
and what it contradicts, and a human answer can resolve the contradiction
without resealing anything: the plan, its progress and the phase all survive,
and the answer outranks the plan where they differ. Turning every ambiguity
into a replan would put the planner above the person who set the goal.
The backward edge is Orchestra's alone. CanReopenPhase is separate from
CanTransitionPhase, which every path validating an agent's request uses, so
phase-request.json still refuses a move back. An agent asks by reporting a
mismatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A detailed plan that nothing enforces is a document. This makes the phases
executable: the implementer may write exactly one status, and every other
status is a conclusion Orchestra reaches by running the plan's own commands.
agent may request: ready_for_verification
agent may not assert: verified, awaiting_manual_verification, failed, skipped
The worker resolves commands from the coordinator, never from the request, so a
request cannot smuggle in a command the planner did not write. They run as argv
through exec with Dir set to the worktree, which is the quality gate's existing
envelope and not a weaker one. There is no shell, so a pipe is a literal
argument.
Project policy decides executable reach. registry.Project.Verification matches
argv positionally, and an absent policy refuses everything: a plan command is
agent-authored, so inheriting the operator-authored gate's reach by default
would be the wrong direction to fail in. A refused command is refused before
anything runs, and the refusal names the project and the command so the planner
learns its real reach.
Two bindings make the record mean something later. PlanRef, so progress earned
under plan A cannot survive into plan B. AtSHA, so "verified" does not outlive
the code that made it true: a record whose commit has moved is retained as
provenance and rendered as stale, never as a claim about the current tree.
Both are the same failure this codebase already fixed for reviews, which bind
to the commit they examined.
Manual steps hold a phase at awaiting_manual_verification. The sign-off is an
ordinary human decision whose subject carries the plan ref and the phase id, so
a later "looks good" on an unrelated thread cannot satisfy a gate nobody was
discussing.
A plan sealed before plan.md declares no executable unit, and says so: the
implement context states that phase progress is unavailable and the work
continues under the old semantics. Inventing phases it never had would be worse
than admitting it has none.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The plan artifact was Changes{Target,Intent} plus three string lists, every
entry capped at 500 single-line characters. That bound makes a specification
impossible: a phase cannot carry a code block, a paragraph of reasoning, or a
verification command with its own argument list. renderSealed then flattened
what little survived through collapse(), so an implement session received a
summary of a summary.
plan.md replaces it. Markdown, 128 KiB, no per-line cap, sealed through the
existing path under the existing PlanRef. The parser enforces the structure the
brief states: required sections, phases numbered from 1 with no gaps, Files,
Changes and Verification per phase, and at least one automated or manual check,
because a phase nobody can verify can never be established as done. Automated
entries are JSON argv arrays, so a pipe is a literal argument rather than an
operator. Headings inside fenced blocks are content, so a plan may show
markdown without parsing its own example.
Citations resolve at seal time against the accepted research, on the
coordinator, which is the only party holding ResearchRef. A plan resting on a
finding nobody recorded fails on the planner while its session is still alive
to be told.
The plan now renders byte for byte into the implement launch, and a rotated
successor receives the same complete document. That is the property the whole
change exists for. collapse() stays for research findings, which really are
short claims.
DecodeStoredPlan reads pre-markdown refs and renders them into the same type,
labelled, so nothing downstream branches on which era a plan came from. A
legacy plan carries no phases, which is honest: the old artifact never named an
executable unit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The brief at agentctx.go:167 advertised findings[].id and findings[].confidence
to every research session. The struct carried neither, so encoding/json dropped
both on every seal, silently, for as long as the schema has existed. A plan
phase had nothing stable to cite and no way to tell an observation from an
assumption.
Finding gains ID and Confidence. Ids are unique within an artifact and shaped
so "research:<id>" is unambiguous in plan prose. Confidence is fact, inference,
or assumption, matching the labels the output style already uses.
DecodeStoredResearch reads what is already in the CAS and backfills both.
Refusing an artifact sealed before this change would block every task whose
research predates it, including at rotation, where the agent that could fix it
is already gone. A backfilled finding is labelled inference rather than fact:
the old schema required evidence and made no verification claim, so upgrading
it on the way in would be the same class of lie this commit removes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The worktree survives a changes-requested round trip, so .orchestra/review.json
from the first review is still there when the second one starts. A reviewer
that writes .orchestra/done without rewriting it would have the earlier
findings sealed against the new commit, and submit binds whatever it reads to
the commit being submitted, so a stale pass is indistinguishable from a fresh
one.
Observed on the 2026-08-28 baseline run: the file from 10:55:45 was still
present when the second review session launched at 10:57:48. That reviewer did
rewrite it, so the run is sound, but nothing enforced it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The loop that reads merged pull requests was guarded by len(pullRequests) > 0
at a point 600 lines before the Gitea wiring that writes to that map. The
length was always zero, so the goroutine never started and a merged pull
request could never complete its task. Live on the first submission this
deployment made: PR #8 took a trusted comment and nothing moved.
Moving the block below the wiring fixes the ordering and avoids the race that
reading the map inside the tick would have introduced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The issues endpoint returns pull requests alongside issues, and nothing
filtered them. The first submission this deployment ever made, kami/test-e2e#8,
came straight back as task 06G4E83E4KRXM8DS90M2648MGM with the submission
packet as its description. That task would have implemented, reviewed and
submitted again, opening a pull request per cycle.
The webhook had the same hole from the other side: a pull_request delivery
leaves the issue key empty, so it would have appended a task numbered 0 with
no title. Both routes now refuse a pull request, and the poll count reports
what was ingested rather than what was listed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Submission needs a sealed review, so a reviewing session that writes
.orchestra/done without .orchestra/review.json refuses every five seconds
with the failure recorded only in worker health. That is the silent loop this
codebase keeps producing: F39 at a phase boundary, F42 at the result commit,
and now the completion tail.
A review-artifact refusal is separated from every other submission failure,
delivered to the pane that can fix it, and clears the done marker so the
corrected file is what finishes the phase.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
The completion tail ended at TaskCompleted with no pull request. Nothing in
the running system ever called the review or submission endpoints: the whole
event log holds zero ReviewRecorded and zero TaskSubmitted, so the merge
reflection, the publisher and the human trust boundary had no entry point.
Four links, in the order the tail needs them:
- finalize commits first and runs the quality gate against the committed
tree, so the gate result is bound to the commit being submitted.
CheckSubmission requires gate sha, review sha and head sha to be one
commit, which a gate run on the pre-commit tree can never satisfy.
- The worker seals the reviewer's findings and submits, through a new
/v1/federation/workers/<id>/submit. A blocking review returns the task to
implementation instead; a project with no forge still completes directly.
- The reviewing session is told where findings go. The brief asked for
findings and named no file, and it described a diff nobody supplied.
- GiteaPublisher.Push asks the forge what the branch holds before reaching
for a local checkout. A worker-owned worktree is on another machine and
has already pushed the commit; the coordinator has no such directory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Ledger current through F42. Handoff records the two defects found in the
completion tail, the deployed revisions, and the two links the tail still
cannot reach because test-e2e has no gate and no delivery configuration.
PLAN-SPEC-DESIGN.md is the operator-approved design for the ACE-FCA plan gap,
committed as a note. No implementation yet, by instruction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F42, live on run 5's completion tail. The review agent wrote .orchestra/done,
the worker recognised it, confirmed the agent idle, and then failed the result
commit on every attempt:
stage result: The following paths are ignored by one of your .gitignore
files: .orchestra/done
.orchestra carries a .gitignore of "*" (internal/herdr/adapter.go:132), so the
marker is ignored, and git refuses an add whose pathspec names an ignored path.
The exclusion now names the directory. Reproduced against git 2.55.0 in a
scratch repo both ways, and the regression test uses the same constant the
worker passes to git.
The failure retried every five seconds for 22 minutes with the task stuck in
review and nothing observable outside the journal, because each identical error
overwrote the single last_error slot. That is F18, still open.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
F41, live on run 5: the review agent produced nothing after 02:12 and the
02:46 renewal was granted anyway. The progress digest covered Claude Code's
status footer, and one of its fields ticked inside the window. Reproduced the
worker's stored progress_sha byte for byte from the live pane, so the branch
taken was progress != ProgressSHA, not IsBusy and not an empty baseline.
PaneProgress now cuts from the editor's lower rule and trims the spinner
summary and version notice above it. AgentStatus still reads the raw capture,
so the busy markers living in the footer are unaffected.
Lease TTL moves to 5 minutes, from domain.LeaseTTL, with renewal at half of
it. Reclaiming a stalled pane happens only at expiry, and 30 minutes per
window made run 5's stall unbounded in practice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
Both halves are on bcaf0cc. The review lease renewed once to 22:56:10Z, since
the agent's 02:12 output was real progress against its launch baseline, so the
recovery lands twenty minutes later than the handoff first said.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Run 5 proved frame through review with four autonomous rotations, sealed
artifacts, and an operator correction rendered above both accepted artifacts and
carried into shipped code.
It is parked on F40, which is fixed but needs the worker installed. Recovery is
the lease expiring at 22:36:06Z and review relaunching with the corrected brief.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
The worker finalises a task when .orchestra/done appears. No brief ever named
that file: grepping a rendered launch.md for it returned nothing, in any phase.
review is terminal. Its only legal move is backwards to implement, so a review
that passes has nothing to ask for and, until now, nothing to write either. Run
5 halted exactly there after four clean rotations, with no error anywhere,
because stopping was the correct reading of its instructions.
The brief now names the marker in the terminal phase, says when to write it,
and says it is exclusive with asking to go back. The test asserts the negative
too, so a phase that can still ask is never told to finish instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Frame to implement, four phase transitions, two artifact refusals the agent
corrected itself, and an operator correction delivered to a live plan session
in three seconds and rendered above both accepted artifacts.
Durable human authority, phase isolation, disposable sessions and reliable
handoff are now observed rather than argued.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Two halves of the same failure, live on run 5.
F38: the phase brief named .orchestra/research.json and described its contents
in prose, never its schema. The agent guessed dead_ends as strings where the
decoder wants {tried, why_failed} objects. The brief now carries the shape, and
a test decodes each documented shape with the same function the worker uses, so
a struct change that is not mirrored fails the build.
F39: the local artifact check refused the request through recordError alone.
answerRefusedPhase only ran on a coordinator 409, so a decode failure told the
agent nothing. The session sat at a boundary rewriting nothing, which is the
silent-loop shape the comment above that block warns about, reached by the one
path with no delivery. Both local refusals now reach the agent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Handoff produced, validated, anchored, committed, predecessor slot freed,
successor leased and started with pickup validated. Autonomous.
F33 and F34 proved themselves incidentally in the same event.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Handoff.Command is the last command observed in the pane. The handoff prompt
tells the agent to write .orchestra-handoff-report.md and stop, so that write
is almost always the last command there. Validate then rejects it as circular:
"must not point to a handoff or report".
Every phase rotation therefore failed on Orchestra's own instruction. Live on
run 5, one stage past F36.
lastObservedCommand now skips commands the validator would call circular, so
the successor gets the last command that was real work. The rule stays in
continuity and is asked, not restated, which is how F36 happened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Run 5 reached the phase transition, further than any previous run, then stopped
on F36: the adapter emits a rotation reason the validator refuses.
Each defect tonight was reachable only after the previous fix. Tests passed
throughout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
handoffReason has emitted "phase_changed" since phase rotations landed. The
continuity validator's reason list was never extended, so every phase rotation
built a handoff it then refused as "invalid handoff meta".
Live on run 5: with F31 clearing the parse ahead of it, the release reached
this and stopped at phase "prepared" with
"adapter: upload handoff: invalid handoff meta".
The test asserts the property rather than the constant: every reason the
adapter can produce must survive Validate, including its fallback.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Two gaps F33 left behind.
F34: the renewal gate exempts a lease with no baseline, so a pane that opened
and never started got a full free renewal period. That is the exact case the
gate exists to catch, and it happened live at 01:33:37: the stuck run 5 lease
renewed to 22:03 on a pane that had not moved since 01:13. Baseline the
progress hash at launch, where the pane is already being read.
F35: nothing could re-poke a live pane. Orchestra can put text in an editor and
be wrong about whether it landed, and the only recovery was to destroy the
lease and wait out expiry, roughly an hour. The new "resubmit" action presses
Enter on text Orchestra itself submitted. It changes no lifecycle state so it
appends no event, and it is fenced like an approval: a live worker-owned pane
at the capture revision the operator was looking at.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Run 5's first attempt never started: ConfirmInput's first poll beat the TUI's
render, saw an empty editor and called it editor_cleared. The prompt sat unsent
while the lease renewed against a pane doing nothing.
Recovery is left to lease expiry rather than a keystroke, so the relaunch stays
autonomous evidence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
ConfirmInput's first poll can run before the TUI renders the pasted text. It
then finds an empty editor, falls through to the idle branch, and reports
confirmation=editor_cleared. The launch text sits unsent for the whole lease
while the coordinator believes the agent is working.
Run 5 died on exactly that: first_submit_at 21:13:19.570, confirmed_at
21:13:19.577. Seven milliseconds. The two launches that worked took ~500ms and
a second Enter, so the difference was scheduling luck.
An empty editor is only proof once it has held the text, or once it has stayed
empty past a settle window. Text seen and then gone still confirms at once, and
so do busy, blocked and queued. Waiting only happens in the never-observed
case, which is the one that cannot be told apart from a slow render.
With this, the failing case reaches the existing resubmit path instead: the
text appears, is recognised as unsubmitted, and Enter is resent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
F29 and F30 are both live-proven on the new build. F30's session cleanup then
exposed F32: a running but empty tmux server answers "no current target", which
hasSession did not recognise, so an already-dead pane could never be reaped.
F32 is a side effect of dea56e4. The reply only exists because the runtime now
outlives its last pane.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
A running but empty server answers "no current target". hasSession only knew
"can't find session", "no server running" and "no sessions", so it returned
that as a real error. Kill then failed for a pane that was already gone, the
quarantine never cleared, and the worker's only session slot stayed pinned.
This is a side effect of dea56e4. Before the runtime became its own unit the
server exited with its last session and answered "no server running", which
hasSession already handled, so this reply had never been produced. AgentStatus
shares the same helper and now reports "exited" instead of erroring.
Found live: the F28 disposable task's session survived its own block on
93338b7, quarantined and unreapable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Runtime death is live-proven distinct from worker deployment: no renewal in the
20:45 window, expiry and requeue at 20:55:50.
F31 explains run 4 and the F28 disposable task both dying at rotation. The
handoff action is two agent-written lines joined by the code, held to one
line's budget, and the error named a branch the answer cannot reach.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
parseHandoffAnswer joins the agent's NEXT and WHY answers with " — ", and the
prompt asks for a sentence each without naming any budget. Validate then held
that join to one authored line's 200 characters. Two ordinary sentences do not
fit, so every rotation failed.
The failure was invisible twice over. The message said "prose smuggled into
list", which named a branch the answer cannot reach: parseHandoffAnswer splits
on newlines and trims, so no authored field ever contains "\n#". The only
reachable cause was length, and the agent was never told what to shorten.
Seen live on two tasks, and it left the release transaction stuck at "prepared"
that pinned workpc-claude's only session slot (F30).
Give Action the budget of both lines, name the length in the error, and put the
limit in the prompt the agent actually reads.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
F28's worker-restart direction is live-proven: three restarts, one pane pid,
one lease epoch, no lifecycle events appended. The runtime-death direction is
still running and will be appended when it lands.
F29 and F30 are both fixed and both unbuilt. Run 5 needs a deployment carrying
them, or it hits F30 the first time a handoff artifact is rejected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
A release transaction that never reached anchor_pushed has no artifact: tx.Ref
is empty and no successor can pick anything up. The event handler still kept
its session mapping alive on TaskReleased/TaskBlocked/TaskFailed, so once the
pane was gone the mapping was immortal. health() reports ActiveTask straight
out of w.sessions, so the coordinator saw the harness as permanently busy and
never leased to it again. It produced no log line at all.
Found live on workpc-claude, stuck at phase "prepared" behind a rejected
handoff artifact. Freeing it needed hand surgery on the worker's state file.
Keep the mapping only while an anchor actually exists. Drop the transaction
with it, since nothing can advance it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
The store fences TaskReleased, TaskBlocked and TaskCompleted on a leased task
against the live harness_id and lease_epoch. The UI action handler sent
neither, so all three returned 409 on exactly the tasks the UI listed them as
enabled for. Found live: eight block attempts against a stuck run at a stable
version, all 409 "task version conflict".
The fence is there to reject a stale writer, not the operator. Carry the lease
read at the top of the handler. The version CAS on the append still rejects a
racing write.
Also initialise body when the request carries none. The block path wrote
block_reason into a nil map.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Type=forking with no PIDFile left MainPID=0, so systemd read a crashed tmux
server as a clean exit and Restart=on-failure never fired. tmux -D runs the
server in the foreground, which gives systemd the real pid.
-D also turns exit-empty off, so the synthetic orchestra-runtime session is no
longer needed to hold the server open past its last agent pane. Nothing else
in the tree referenced that session.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
The socket is /tmp/tmux-$UID, so orchestra-tmux.service and
orchestra-worker.service must run as the same user or the worker talks to a
different server. The installed unit has run as kami since before the split;
the repo copy still said orchestra.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
F28. The worker spawns the tmux server on its first command, so the server and
every agent pane sit in the worker unit's cgroup. Restarting the worker
destroyed the sessions it was restarting to manage, and F16's missing-pane
branch has been firing on deployment rather than on real execution loss.
KillMode is not the fix. Under mixed systemd still SIGKILLs the cgroup
remainder once the main process exits, and process only encodes accidental
orphaning. The runtime becomes its own service instead.
The worker gains After= and Wants= on it, ordering only: a worker that finds
the runtime missing must report that rather than be stopped by it. The unit
holds an idle session so the server outlives its last agent pane.
User must match between the units, since the socket lives under /tmp/tmux-$UID.
The installed worker on workpc runs as kami while this file still says
orchestra; the staged copy is set to kami to match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
F25. rotationTick returned early for claude before reaching federatedTurn,
which has one call site below that return. On the harness both burn-in runs
used, no phase request could ever be read and every human decision recorded
against a live session went undelivered. Claude still skips the occupancy
state machine below, because it owns its context rollover through the
installed hook. A turn boundary is not a rotation.
F26. The phase brief listed every domain-legal target, so run 4's frame
session read "research, implement" and asked for implement, which the
project's path refuses. The path is Orchestra's to know: the brief now names
one step and says a wrong target comes back with the right one.
F27. A refused request only reached recordError, leaving the agent to rewrite
the same rejected file forever with nothing telling it why. federation.
StatusError makes a 409 classifiable, and the refusal is delivered through
sendPrompt under the F20 guarantee. A transport failure is not an answer: the
request survives and the agent is told nothing.
The F25 regression test fails against the unfixed rotationTick.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
F24 is the last Orchestra-owned editor write outside the delivery guarantee,
in the coordinator-local CLIAdapter.LeasePrompt. It cannot execute on the
deployed topology, so it is logged rather than fixed: run 4 exercises the
federated worker path, whose launch delivery already confirms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
The brief told the agent to ask for a phase change and never carried the
asking. The agent asked in prose, no code represented the request, and the
session idled until its lease expired. That is what failed run 3.
F21. The agent asks with .orchestra/phase-request.json, and seals
research.json or plan.json where the phase it is leaving produces one. At a
verified turn boundary the worker checks the phase belief, the transition and
the artifact, then calls the coordinator with its lease epoch and a derived
operation id. AdvanceWorkPhase is unchanged, so a request cannot reach a move
the operator surface could not also make. Redelivery is idempotent.
F22. A session now records the phase it was launched to run. One that no
longer matches its task rotates with reason phase_changed, whether this worker
asked for the change or an operator made it.
F20. CLIAdapter.prompt sent handoff and rotation prompts without confirming
them, which is the failure F20 exists to catch. Fixed at the shared call site.
F23 needed no change. Issue comments already become decisions with no
submission, through Reconciler.Reconcile at PreLease and at every turn
boundary. The earlier finding searched internal/operations alone and was
wrong. Tests now cover the boundary it turns on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
Run 3 is waiting on an autonomous expiry and relaunch, and the next session
has to judge it without repeating this one mistake: F16 was reported as
passing live on the strength of renewal events, while its predicate was
inert. The hierarchy the operator set is at the top of the file, and the two
hard failure predicates are stated before anything else.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
F20. Only the launch confirmed its submit. A decision notice at a turn
boundary, and /clear or @HANDOFF.md during a context reset, were
fire-and-forget through the same transport that loses an Enter often enough
that the launch needed three resubmits. A lost Enter on the context-reset path
is the worst of them: it strands the session mid-rollover and nothing retries
it. LaunchConfirmer is therefore InputConfirmer, ConfirmLaunch is ConfirmInput,
and sendPrompt and sendLine both go through it.
Orchestra does not try to guarantee delivery of input it did not originate.
But it must never read that input as work, which is the F16 half. Burn-in run
3 stalled with an unexplained "go ahead and implement it" in the editor, and
the renewal check hashed the whole capture, so those keystrokes read as
progress and the lease kept renewing around an idle agent. PaneProgress drops
input lines from the capture, which the -J join makes exact: a wrapped input
block is one line beginning with the prompt marker.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
F16 shipped inert. renewLeases refuses to renew when the agent is idle and the
pane is unchanged since the hash recorded at the previous renewal, but
reconcileLeases rebuilt every lease from the coordinator's task list on each
tick and constructed a fresh struct, dropping ProgressSHA. The check therefore
took its no-baseline branch forever and renewed unconditionally.
Observed live: task 06G44JZB80MZBEY97196EZN8EC renewed at 09:31, 09:52 and
10:12 while its agent had been idle since 09:13, with a byte-identical pane
capture and no busy marker anywhere in it.
UsageBaseline and PickupAcknowledged were being wiped the same way, so a usage
receipt lost its baseline and a pickup could be re-acknowledged. The
coordinator is authoritative for the lease, not for what this worker observed
under it, so those three carry across a reconcile while the epoch is unchanged
and reset when it is not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
F12 through F19, each against the revision it landed in, with what was
verified live and what has tests only. AUDIT.md is uncommitted and owned by
another session, so the ledger goes here rather than colliding with it.
The entry worth keeping: run 3 failed three times with prompt_not_submitted
while six isolated probes of the same code path submitted on the first Enter.
F17's evidence line settles it, submit_attempts=2 in production against 1 in
every probe.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A task that reached the router's MaxAttempts was permanently terminal.
TaskReleased only ever increments Attempt, TaskCorrected could not touch it,
and no HTTP route emitted a correction at all. The only way to work an
exhausted issue again was to invent a second task for it, which defeats
(source, external_id) dedupe and abandons the task's own history.
POST /v1/tasks/{id}/retry, full-control surfaces only. It requires the task
to be failed, unleased, and failed with reason retry_limit: restoring a retry
budget is not an answer to a failure that was not the budget running out. The
effect is one TaskCorrected naming that failure, setting state queued and
attempt 0 and clearing next_retry_at, failure_class and last_error. Task id,
source pair, goal, acceptance, decisions, work phase and artifact refs all
stay, and the original failure events stay in the log.
operation_id is required and makes the call idempotent, so a repeated request
cannot reset an attempt that has since started running.
This is RetryTask, not a generic correction endpoint: arbitrary task mutation
over HTTP is a different and much larger authority. It also does not address
F9, which is an operator releasing a lease someone else owns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Burn-in run 3 failed three times with prompt_not_submitted: the launch text
reached the editor every attempt and the following Enter never took effect.
Six isolated probes of the same code path, environment and worktree all
submitted on the first Enter, so the submit is not deterministic and waiting
for it to land is not enough.
F17 first. pendingInput scanned every line beginning with the prompt marker,
but a queued or already-accepted message renders with the same prefix. Only
the editor that owns the pane cursor is unsubmitted input, so inputState asks
tmux for the cursor row and reads the editor around it, joining soft-wrapped
rows.
On that footing ConfirmLaunch becomes an active submit protocol: resend Enter
while the live editor still holds exactly what was submitted, at most three
times and no closer together than two poll intervals, then observe until the
deadline. Queued input confirms rather than fails. The evidence records
confirmation kind, submit_attempts and both timestamps, so a harness that
needs a second Enter is distinguishable from one that needs none.
Verified live against a real Claude Code pane: confirmation=editor_cleared
submit_attempts=1, no spurious resend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
renewLeases renewed whenever a session existed and PaneCapture succeeded,
so a pane that opened and never accepted a prompt held its lease forever.
That is the mechanism behind the July stuck task: the launch failed and
nothing ever let go.
Renewal now needs the agent to be busy, or the pane capture to differ from
the one recorded at the previous renewal. The first renewal has no baseline,
so it records one and passes; the next must show movement.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Run 2 never reached conformance. It found F12, then F15, then F16, each
hidden behind the one before it. F12, F13, F14 and F15 are fixed and
committed; F16 is diagnosed and open. Neither deployed half is at 1fd82f8
yet, and the worker install needs root, so the next session starts with an
operator step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Burn-in run 2 recorded TaskLaunchAcknowledged, opened a pane, and ran nothing
for fifteen minutes. The launch instruction sat in Claude Code's input editor
as "[Pasted text #1 +66 lines]" at zero tokens and zero elapsed. Two separate
bugs produced that.
The transport was wrong for the harness. TmuxBackend.Prompt writes the whole
instruction with send-keys -l and then sends Enter, and the TUI coalesces the
fast multi-line write into a paste that absorbs the following Enter. Launch
transport is now a backend property rather than one universal prompt format:
claude on tmux submits a single line pointing at .orchestra/launch.md, every
other harness keeps the inline path it was verified on. agentctx is unchanged
and the file still holds the exact bytes Orchestra rendered, so what the agent
receives is identical either way. Under the file transport a failed write is
now a failed launch, because there the file is the instruction.
The acknowledgement was also wrong. It meant "Prompt returned nil", not "the
harness accepted the prompt". Backends may now implement ConfirmLaunch, and
the tmux one polls until the input editor clears and the agent is observably
busy, blocked on approval, or at least no longer holding the text. An editor
that still holds the prompt at the deadline is a definite failure. The worker
kills the pane, drops the session so the retry starts clean, and returns
ErrPromptNotSubmitted, which classifies as prompt_not_submitted rather than
launch_uncertain. That class already falls through to TaskReleased, so the
existing retry path takes it and no lease is held on a launch that never
happened.
The confirmation bound is tunable because how fast a terminal harness reacts
is a property of the host. It is not a sleep before the submit: the submit is
deterministic, and this waits for the harness to visibly react to it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gitea ingest set title, description and capability but never acceptance, so
every Gitea-sourced task rendered "Acceptance: Not stated." however carefully
its body was written. F2 restored the body; it did not wire the field.
The convention is deliberately tiny. A markdown heading spelled "acceptance"
or "acceptance criteria" opens the section, bullet and checklist items until
the next heading become the criteria in order, checkbox markers are stripped,
empty items are dropped, and the whole section leaves the description so a
criterion is never also read as instruction prose. Nothing else is recognized:
inferring acceptance from arbitrary prose would eventually invent a
requirement, and a fabricated criterion outranks every human decision beneath
it in the authority order.
An absent section yields no acceptance, which is not an ingestion failure. The
task renders "Not stated." and the frame phase resolves it through the
decision-request path, where a human answers instead of the parser guessing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Burn-in run 2 ingested its task and then sat queued forever. Every herdr in
the live config declares quota_limit_5h and quota_limit_weekly, the event log
holds zero QuotaReported events, and QuotaSince reported an empty window as
unknown. QuotaAvailability fails closed on unknown, so no harness could ever
be leased, and the only producer of a receipt is a completed lease.
The event log is Orchestra's whole accounting source, so a window holding no
receipts is observable zero consumption. QuotaSince now reports known for an
empty window and for a harness that has never reported. A receipt that
declares its own consumption unknown still fails closed.
The refusal also lied about its cause. federatedAvailability collapsed a base
gate refusal into the federation health string, so router health said "stale
heartbeat or unhealthy local backend" while the heartbeat was one second old.
Availability gates now name themselves through an optional
ReasonedAvailability contract: quota refusals say whether usage is unknown or
the window is exhausted and by how much, and worker refusals distinguish an
unregistered worker, a never-probed backend, a stale heartbeat, a stale
health check, and an unreachable backend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This checkout is shared with another session. Its uncommitted Go changes must
neither be compiled into a binary stamped with a commit revision nor block a
deploy, and a dirty-tree refusal does both jobs badly. deploy/build.sh now
builds in a detached worktree of the revision it stamps, and the container image
is built the same way rather than from the live checkout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The API was in a restart loop, exiting with `invalid event: until_ns required`.
ValidateEvent compared until_ns against time.Now() for TaskLeased and
TaskLeaseRenewed, so a lease event that was valid when written failed validation
once it expired. store.Open replays the log tail after the snapshot and
log.Fatal's on the first invalid event, so the coordinator refused its own
history and could not start.
Validation of a durable event must be time-independent. Well-formedness is this
function's question; freshness belongs to Store.Lease and Store.ExpireLeases,
which compute until_ns themselves.
Latent since the field was introduced. It needed a renewal in the post-snapshot
tail plus a restart after that renewal expired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A task in retry backoff was filtered out before the candidate loop, so it
recorded no rejection at all: queued, apparently assignable, and silent. That is
the exact shape that made F5 take a live session to diagnose. It now reports
"retry backoff until <time>".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
F7, security. An unset surface token makes the middleware skip its check, so a
full-control surface with no credential is an open control plane rather than a
closed one. With ORCHESTRA_TUI_TOKEN unset, any LAN caller could lease, release,
complete or block any task by declaring one header, which is how this session's
manual leases were issued. authz.RequireCredentials now refuses startup instead
of logging. Web is exempt: Sessions makes its login mandatory.
F5, lifecycle. router.go's silent `continue` was the first bug, not the
predicate behind it. Every eligibility gate now records a router.Rejection with
task, herdr and reason, exposed at GET /v1/router/health, reset per pass. No
gate was weakened: a direct Store.Lease succeeding proves the lease path, not
that eligibility should have selected that worker.
F8, correctness. Reconcile iterated every configured source for every task, so a
task's external id was looked up in whatever repository each source pointed at.
Once two repositories share an issue number, an unrelated human comment becomes
an authoritative decision for the wrong task. Reconciliation is now bound to
task.Source, the provider:project identity the ingest stamped, and a source that
cannot prove it owns the task is skipped. A task with no matching source
reconciles to nothing and still launches, because nothing to import is not a
failure to read.
The integration fixture ingested from "jsonl" while reconciling from "gitea",
which is exactly the shape F8 makes impossible; it now ingests from the source
it reconciles.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The .orchestra directory showed up as untracked work in the very worktree whose
own launch context said "uncommitted changes: false". It would have polluted
the quality gate, the review diff, and the agent's reading of git status. A
.gitignore of "*" inside the directory ignores it including itself. The
worker's done marker lives there too and had the same problem.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by reading .orchestra/launch.md on the first burn-in task. The rendered
context said `Acceptance: Not stated.` and carried the issue title as the whole
goal, because Gitea.event parsed the issue body and then dropped it from the
TaskCreated payload.
The body is the task's own statement of what it wants, which every rendered
context ranks above continuity and below only a human decision. The store
already reads `description` from TaskCreated; only the provider was silent.
Every Gitea-sourced task so far has therefore run on its title alone. Classify
as an authority bug, not a model-following failure: no agent could have known
what it was not told.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found live on the first burn-in task. Every federated launch failed with
`effective intent: federation: 401 Unauthorized: unauthorized surface`, and the
task nacked back to queued.
GET /v1/tasks/<id>/intent was added so a worker renders its own launch
instruction from the reduced intent, but authz.HTTPWithSessions never got the
matching worker-path exemption. An unlabelled request defaults to the Web
surface, which is session-gated, so the one caller the endpoint exists for could
never reach it.
Twenty test packages passed throughout, because the authz tests covered the
surface capability table and not the worker path list. Only the live owner path
established this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The burn-in pairs a coordinator with a worker, and step one of every run is
checking that both report the burn-in commit. That was only visible behind the
operator login, at /v1/admin/diagnostics and /v1/federation/workers, and the
worker never logged its own revision at all. Both now print it on startup, so
`docker logs orchestra-api` and `journalctl -u orchestra-worker` are enough.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v3 stack, previously an uncommitted working tree, plus this session's two
units and the burn-in instrument. This commit is the burn-in build identity:
coordinator and worker must both report this revision before a task is created.
Workflow (earlier sessions, uncommitted until now): human decision events and
reduction, source cursors and reconcile-before-launch, turn-boundary
reconciliation, internal/agentctx as the single renderer, ace-fca phases with
sealed artifacts, the trajectory gate, bounded grilling, independent review,
task pr enforcement, and human review reflection.
Capability restrictions at the agent boundary: an authz.Agent surface at
GatedWrite may ask and may not act. It also fixes two bugs the unit exposed --
gated surfaces could not reach the two endpoints written for them, and
RequestHumanDecision would block an unowned task while rejecting a question
from the session that did own it.
Turn-boundary reconcile-failure escalation: a streak of consecutive failures
asks the session to hand off, fenced on the lease epoch, with reconcile_failure
as a real handoff reason. The worker was dropping the coordinator's verdict on
the floor; it now acts on it.
Burn-in: herdr.WriteLaunchContext dumps the exact agentctx.Build result to
<worktree>/.orchestra/launch.md at every launch, local and federated. BURNIN.md
is the runbook. deploy/build.sh stamps both binaries from one commit.
go build, go vet and go test ./... pass, 20 packages.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Acts on the seven review comments on PR #1.
Design A is gone (comment 4). clients/ deleted rather than tracked: with
workers carrying cross-machine work the bridge is undeployed, which supersedes
the 2026-07-27 "keep through Phase 5" decision. CLAUDE.md, AGENTS.md and
AUDIT.md updated from "retained" to "deleted".
The harness-hook completion path is gone (comment 10). Investigation of the
live OpenCode QA run showed orchestra-worker owns completion end to end: it
watches for .orchestra/done, confirms via AgentStatus that the agent is not
busy, then posts through /v1/federation/* with both lease epoch and expected
version. The hook scripts used a different, older convention
(.orchestra-report.md) and posted to /v1/harness/complete, which had already
been reduced to a 410 stub - so that path could not have completed a task.
Nothing exercised it, because the live run never used it. Deleted: the three
deploy/hooks scripts, the 410 route, the unmounted harnessCompletion handler,
and its test. That test passed against a handler no mux routed to, which is
the exact "looks wired but isn't" pattern CLAUDE.md warns about; the
constant-time token compare added to it earlier today goes with it, having
never been reachable. /v1/harness/turn is untouched and still live.
Retired deployment files (comments 8, 12, 14): deploy/orchestra.service and
deploy/redeploy.sh (which sudo-installed to /usr/local/bin and restarted that
unit), plus deploy/docker-api-entrypoint.sh. The entrypoint was safe to remove
once its premise was checked: env vars reach the container through
`env_file: .env` in compose.yaml, not by sourcing /etc/orchestra/orchestra.env
- only config.jsonc is bind-mounted there - and Dockerfile.api's line 17
already sets ORCHESTRA_DATA/ORCHESTRA_PORT. Dockerfile.api now execs
/app/orchestra directly. orchestra-worker.service is a different, current unit
and is kept.
deploy/config.example.json deleted as a duplicate (comment 6); the annotated
.jsonc is the one registry.go points at, and its header no longer tells the
reader to copy the file that just went away.
Documentation corrected beyond the deletions:
- CLAUDE.md's deployment section claimed the container bind-mounts
/etc/orchestra:ro and its entrypoint sources the env file. Both wrong.
- AGENTS.md still described a systemd deployment on homesrv as of 2026-07-27.
- AUDIT.md's H5 row still described a "retained compatibility handler".
- deploy/DEPLOYMENT.md still named redeploy.sh as the deployment path.
- deploy/orchestra.env.example still cited EnvironmentFile=.
TOKEN_MINIMAL_WORKFLOW_PLAN.md (comment 2) is untouched: it and WEB_UI_PLAN.md
were both missed by REVIEW.md's documentation sweep, and reconciling a 534-line
forward-looking plan against AUDIT.md is its own task, not a review fixup.
Verified: go build ./..., go vet ./..., go test ./... all pass after the
deletions, and go list ./... has no node_modules entry. No live herdr or pane
was touched; nothing was deployed. The running image still predates this
commit until compose is rebuilt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GEugbHVYfAXFpTqDYbByEB
Acts on the 2026-07-30 senior review (REVIEW.md findings 1, 2, 4, 5, 7).
Docs (finding 1): CLAUDE.md and AGENTS.md both claimed Design B "has zero
clients - no worker binary exists". cmd/orchestra-worker/main.go is the
deployed worker, and the non-local-herdr guardrail has landed in
Coordinator.adapterFor. Both sections rewritten; AUDIT.md gains a matching
federation-status record. The Phase 5 retention / Phase 6 deletion decision
for Design A is preserved, not flattened.
clients/ un-ignored and tracked, including the .service unit and README:
deployed code belongs in version control. Design A is NOT deleted here.
progress.md (finding 2): the file was deleted after 636ed8a, yet CLAUDE.md
instructed every session to cross-check against it. References removed from
CLAUDE.md, AGENTS.md, internal/orchestrator/rotation_test.go (comment only)
and deploy/hooks/orchestra-codex-poll.sh; AUDIT.md now carries the log role.
web/go.mod (finding 4): a module stub ends the parent package graph at the
directory boundary, so go list ./... no longer yields
web/node_modules/flatted/golang/pkg/flatted. A build tag cannot work - the
package is in the package list before tags are evaluated. Local/CI-only
breakage: Dockerfile.api builds ./cmd/orchestra by explicit path and
.dockerignore already excluded node_modules.
orchestra-worker (finding 5): untracked (8.9MB, mode 100755, still on disk);
both binaries now gitignored.
Token compare (finding 7): cmd/orchestra/main.go:139,582 use
subtle.ConstantTimeCompare, matching the authz.go idiom. The token != ""
guard stays first, so an empty configured token still means auth-disabled
rather than auth-bypass. Three further plain != secret compares remain in
internal/federation/federation.go:343,346,368 - tracked, not fixed here.
Also included from the review pass: orchestrator.go records adapter-resolution
failures in SessionHealth.LastError instead of dropping them on a bare
continue, plus an Observed flag so lease-seeded health is not mistaken for a
live reading, with a covering test. GET /v1/tasks/<id>/health now returns a
record with last_error where it previously returned a bare 404.
REVIEW.md's own second pass claimed every checkable fact held up; four did
not. AUDIT.md never contained the false Design B claim (AGENTS.md was the
second copy), the guardrail is at orchestrator.go:312 not :309, the
progress.md site list missed the codex-poll hook, and only orchestra-worker
was tracked. Verified: go build, go vet, go test, and
go list ./... | grep node_modules all clean with every change applied
together. No live herdr or pane was touched; nothing was deployed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GEugbHVYfAXFpTqDYbByEB
The deployment moved from orchestra.service to Docker Compose, but both
CLAUDE.md and AUDIT.md still described the systemd unit. A stopped unit
plus a stale /usr/local/bin/orchestra reads exactly like a dead service,
so document where the live deployment actually is and that deploying now
means rebuilding the compose images.
Also record what a live check found: the running image predates the B18
auth commit, all six herdrs are unreachable (the sole remaining blocker
on live proof for B13-B17), GET /v1/tasks has no authz gate but sits
behind a ufw-restricted bind, and ntfy still 403s.
Sharpen the herdr-logging note: successes are never logged, so a herdr
with no log line is up, not down, and the logs can't confirm one coming
back. Probe the ports directly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
All five defects filed while implementing B18, plus the router flake that
predated them. None of this has run on the deployed instance: the service
is stopped and /usr/local/bin/orchestra predates every change here.
B20 is the one that could silently defeat approvals. The capture revision
was UnixNano, so it changed on every read and said nothing about whether
the pane had changed; it is now an FNV-1a hash of the pane text, changing
iff the text does. The worse half was precedence: capture() preferred the
coordinator over a published worker capture, handing Queue a timestamp the
owning worker's staleness check could never match, so every federated
approval resolved "stale" and the keystroke never happened. Worker captures
now win — their existence means a registered worker owns that pane — and
capturePane follows the same precedence via Capture.Source rather than
guessing.
B19 was filed as "federated approvals emit no event", which overstated it:
the resolution half already existed, and correctly fires only on an
acknowledged worker report. The missing half was the request. Server.action
now appends ApprovalRequested at queue time, subject_ref set to the command
ID the later resolution carries. If that append fails the queued command is
resolved "rejected" — a keystroke that left no audit trail must not run.
B21 bounds the command list: resolved commands prune after 30 minutes on
both Queue and Commands, pending ones never at any age, since dropping one
would discard an operator decision. The persistence half stays open and is
recorded as such — captures and commands are still in-memory only.
S12 splits ORCHESTRA_NTFY_TOKEN, which was both the secret handed to the
ntfy server and a valid inbound credential for the ntfy surface; the latter
is now ORCHESTRA_NTFY_SURFACE_TOKEN. Breaking: a deployment relying on the
old dual use has no inbound gate until it sets the new variable. S13
deletes the dead auth() copy of the authorization policy.
The router flake was in the test, not in assignment. Store.Tasks() ranges a
map, and the assertion indexed two separate Tasks() calls, failing whenever
the orderings disagreed; instrumenting it showed a valid TaskLeased and a
genuinely leased task on every "failing" run. It now snapshots once and
asserts that exactly one task is leased, and passes at -count=60.
AUDIT.md records what is still not done: the deployed env and binary, the
live re-verification B13-B17 has always lacked, and two operational faults
found in the journal that block it — all six herdrs are refusing
connections, and ntfy delivery is failing 403 on every send.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
The UI is a full control plane: it can create tasks, release or complete
them, and inject approval keystrokes into live panes. authz.HTTP did cover
it (an absent surface header defaults to Web), but the gate is opt-in and
the deployed env sets no tokens while binding all interfaces, so in
practice it was reachable unauthenticated from the LAN. Setting the token
alone did not work either: a browser cannot put a bearer token on a
document load, so the UI would 401 on index.html.
- ORCHESTRA_WEB_TOKEN is now mandatory; startup fails rather than silently
serving an open control plane.
- authz.Sessions issues random values stored SHA-256-hashed, with a TTL,
so a leaked snapshot yields nothing usable.
- POST /v1/ui/session verifies the token in constant time and returns it
as an HttpOnly, SameSite=Strict, Secure cookie. This is a presentable
form of the same credential, not a new authority.
- HTTPWithSessions accepts that cookie in place of the bearer token, and
only for the Web surface. The login endpoint and non-/v1/ GETs (the SPA
shell) are exempt by necessity; every /v1/ control path stays gated.
Note this is a breaking config change: .orchestra-config/orchestra.env
sets no tokens, so the service will not start until it does, and setting a
Web token newly gates the other /v1/ surfaces that default to Web.
AUDIT.md is reconciled against the code rather than against itself. B14,
B15 and B16 are closed with their evidence; B17 is closed on the worker
path only; the stale claim that B13 was open is corrected. Adds the
previously undocumented command channel and web UI, and files what that
implementation pass surfaced: federated approvals emit no event (B19), the
local capture revision is a timestamp rather than a change counter and can
silently defeat approvals (B20), the command queue never prunes (B21), the
ntfy token serves two unrelated purposes (S12), and a dead copy of the
authorization policy sits in main.go (S13).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
Introduces the browser-facing surface and the worker-side protocol that
backs it:
- internal/ui: joined read model plus per-task lifecycle and approval
controls, kept separate from the raw endpoints workers and harnesses
depend on.
- internal/webui + web/: Vite/React app, build output embedded via
go:embed and served as an SPA fallback.
- federation: per-(worker, task) captures with a monotonic revision that
advances only when pane text actually changes, and a command queue
restricted to grant_approval / deny_approval, each bound to the capture
revision the operator acted on.
- orchestra-worker: publishes captures and executes commands only after
re-reading the pane and confirming the revision still matches. Sends
keystrokes only for a visible y/n prompt or OpenCode's fully labelled
selector, and refuses to deny through that selector rather than guess
at unobservable navigation.
This is the ownership boundary AUDIT.md's B14 and B17 call for: approval
becomes an explicit, revision-bound operation executed by the worker that
owns the pane, instead of a side effect of prompting over a
coordinator-driven remote socket.
Also ignores the web build inputs and outputs. node_modules ships vendored
Go packages, so go build and go test walk into it if it is merely
untracked; both node_modules and .node_modules are excluded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
herdr hands a freshly created pane/agent back before it's actually ready,
and rejects the very next call with a range of different transient errors
("not an available shell", "not an active named agent", "target ... not
found") depending on timing. String-matching each wording as it turned up
live proved unwinnable across three live redeploy-and-test rounds, so
StartAgent and Prompt now retry any error for up to 15s (bounded by
wall-clock time, not attempt count) rather than pattern-matching herdr's
error text.
Confirmed live against workpc: a fresh lease (wD:p1) now reaches a real
attached claude session instead of failing before the agent starts.
Live testing also exposed a second, separate defect (B13, documented in
AUDIT.md, not fixed here): agent.start can return success while never
actually starting an agent when two leases land close together, with no
error for a retry to catch. Left three test panes on workpc untouched
(wD:p1, wE:p1, wF:p1) pending manual cleanup, per the standing rule against
destructive herdr calls without asking first.
Also folds in the already-flattened AUDIT.md/progress.md merge that was
staged ahead of this session's changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
The previous function_call/function_call_output shape was never verified
and doesn't exist in any real Codex rollout. Confirmed the real shape
against this machine's own ~/.codex/sessions files: file edits arrive as
event_msg/patch_apply_end (changes+success, no pairing needed), and shell
commands arrive as a freeform custom_tool_call named "exec" whose input is
a JS snippet embedding cmd:"..." rather than a flat arguments object, with
failure signaled by a literal "Script error:" prefix in the output text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Closes the last two S11 triggers. internal/herdr/activity.go normalizes
tool/function calls per harness (ClaudeActivity verified against the
existing transcript format, CodexActivity best-effort/unverified,
OpenCodeActivity refuses — no confirmed per-tool-call source exists) and
implements the three thrash rules plus a narrow milestone check
(successful git commit as the last call).
CLIAdapter.RequestHandoffReason asks the agent to write a handoff with
meta.reason set, same "ask, don't invent" pattern as the existing handoff/
report requests. rotate() and TurnDecision generalize the manual-bypass
shortcut to manual/milestone/thrash and request (never directly release)
on a detected trigger.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Implements §3.1's invariant that a wrong event is never edited, only
compensated for by a new appended event. TaskCorrected references the
event it repairs and can change state and/or amend-style fields;
Store.Append verifies the referenced event actually exists on the task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
rotate() and TurnDecision now check the agent's own handoff for
meta.reason=="manual" before evaluating occupancy/turn-boundary — per
spec §5.3, that reason is itself the boundary signal ("a coherent unit
finished and the next is independent"), so it bypasses both checks and
releases immediately. Extracted the shared release-and-anchor-certify
tail into Coordinator.finishRelease so the manual path gets the same
anchor safety guarantee as the threshold path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Coordinator.Soft (default 0.55, ORCHESTRA_OCCUPANCY_SOFT) makes rotate()
and TurnDecision request a handoff advisory-only once occupancy crosses
the soft threshold, well before Hard forces a release/rotation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Register() previously trusted a self-declared id and self-chosen token
from any caller, and let a second caller silently hijack an existing
worker id by re-registering it with a different token. Adds an optional
pre-shared AdmitToken (ORCHESTRA_FEDERATION_ADMIT_TOKEN) and requires a
same-id re-registration to present the existing worker's own token.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
The 1s ticker's own ExpireLeases call almost always won the race against
the coordinator's 30s Monitor loop, so the coordinator's ExpireLeases saw
nothing left to expire and never reached its session-kill path — silently
orphaning herdr panes past their lease TTL whenever a coordinator is
configured. AUDIT.md called this "harmless" on the assumption CAS merely
picked a winner; the real effect is the pane-kill side effect never firing.
Now the ticker defers reclaim entirely to the coordinator when one exists,
keeping only AssignPending as a periodic retry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Adds an optional "harness" field so codex/opencode completions route to
their own Usage readers instead of always assuming Claude's transcript
format; unblocks the server side named as open in progress.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Amendments to due/description/inherent_priority were accepted and durably
logged but silently discarded by the projection since store.apply only
ever handled the title key. Also added the missing Task.Description field
(TaskCreated never populated it either).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Brief.Git was a single GitSync read from ORCHESTRA_DATA (never a git
checkout), and completions were counted but discarded their report_ref/
receipt. Brief.Git is now keyed by project ID and built from each
project's real repo; GitSync gained Ahead/Behind vs upstream; Brief now
carries Receipts pulled from each TaskCompleted payload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Closes S4 (AUDIT.md): Fanout.Run returned on the first sender error,
permanently ending notifications for the process lifetime after one ntfy
hiccup. Failed sends now go through an OnError hook and the loop
continues. Also persists the delivery cursor to a file next to
ORCHESTRA_DATA so a restart resumes from the last delivered event instead
of re-notifying the entire log from seq 0. Adds the package's first test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Closes Phase 2 items 1-2 (AUDIT.md): Coordinator.TurnDecision evaluates
occupancy/turn-boundary/handoff state synchronously per turn and returns
continue/prepare_handoff/rotate_now/refuse, exposed via POST
/v1/harness/turn. The Claude Stop hook now calls it on ordinary turn
boundaries instead of no-op'ing, and exits 2 on refuse.
Also closes B7's post-hoc producer: /v1/harness/complete now appends a
QuotaReported event from the completing lease's harness usage, so the
router's quota-availability filter and the brief's quota_consumed stop
evaluating against a permanent zero. Live per-harness push producers
(Claude statusline, Codex rollout tail) remain unbuilt — investigation
recorded in AUDIT.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
MarkdownChanges was deleted as dead code, but the underlying spec
requirement wasn't abandoned — rebuilt it independently. Adds
continuity.ConventionsHash for AGENTS.md/CLAUDE.md/VOCAB.md, tracks a
per-session snapshot on herdr.Session, and adds
Coordinator.checkConventions (run every Monitor tick) which compares
each active session's snapshot against its project's base repo and
pushes an in-pane notice via a new herdr.ConventionsNotifier
capability when they drift.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Release already validated and uploaded a §6.1 handoff, but nothing ever
told the agent the .orchestra-handoff.json convention existed, so the
file it waited on never got written. rotate() now prompts the agent
once via a new optional herdr.HandoffRequester capability
(CLIAdapter.RequestHandoff) when the file is missing, and defers
Release until it appears, mirroring the .orchestra-report.md/B3 ask
pattern rather than inventing a handoff.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Release now re-verifies every handoff Anchor.Dirty file hash (previously
unchecked after the top-level anchor SHA compare), snapshots dirty state
onto a per-task scratch branch before uploading, and rewrites the anchor
to the new commit so successor pickup collapses to a single HEAD compare.
ScratchCommit made idempotent for repeated rotations of the same task.
Bootstrap's prompt now points the agent at the scratch-branch commit
history instead of vague "read the handoff" prose, and does not claim a
GET /v1/artifacts/<ref> endpoint that doesn't exist.
MarkdownChanges had zero callers and zero tests; deleted per AUDIT.md's
explicit deletion option rather than half-wiring an undesigned feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Fixes AUDIT.md's B6: nothing wrote a TASK.md into a worktree, so
continuity.ValidatePickup had no caller and no file to check.
- continuity.RenderTaskFile/TaskFileHash: render and hash the immutable
§6.2 TASK.md from a domain.Task.
- GitWorktrees.Create writes and commits TASK.md into every freshly
created worktree (must be committed, not dirty, for ScratchCommit's
immutability check and for a stable hash).
- Coordinator.Start now runs continuity.ValidatePickup (anchor SHA,
dirty-file hashes, TASK.md hash) against the real worktree before
bootstrapping a successor onto a handoff_ref, and blocks the task
instead of bootstrapping on a validation failure.
Still open from Phase 4: handoff production (agent writing the real
handoff; Release still refuses per B5), ScratchCommit wiring before
release, and the §6.2 bootstrap-prompt rewrite — see AUDIT.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
CLIAdapter.Lease's initial prompt used wait=0, skipping the inline wait
Bootstrap already used. The spec (§5.1) requires inline wait on
agent.prompt for bootstrap injection so the prompt isn't sent into a
half-rendered pane.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
POST /v1/harness/complete lets a Claude Code Stop hook report task
completion instead of relying on a human hitting the manual endpoint.
The hook only fires on an explicit .orchestra-report.md marker (not
every turn boundary); the server builds the receipt itself from the
real transcript via herdr.ClaudeUsage rather than trusting a
self-reported number. Codex/opencode producers and the turn-decision
endpoint are still unbuilt — see AUDIT.md/progress.md for scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Verified against a live herdr instance (192.168.1.105:9245) that pane.kill,
pane.release, and pane.rotation_signal never existed in the protocol, as
AUDIT.md's B5 suspected. Real method list captured in deploy/herdr-schema.json.
- Kill now calls the real pane.close({pane_id}).
- RotationSignal interface/method/call-site deleted; no real equivalent exists.
- Release now refuses loudly instead of calling a nonexistent method — the
real pane.release_agent can't return a handoff_ref either way (herdr
doesn't write handoffs, the agent does), so a real fix needs Phase 4
handoff production first.
Also documents Phase 0 findings in AUDIT.md/progress.md, and adds
CLAUDE.md/AGENTS.md with project-specific knowledge (herdr protocol facts,
deployment topology, a currently-stuck live task, the federation fork) for
future sessions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
S5: Store.Lease and Store.ExpireLeases both set Event.ID to the task id, so
every TaskLeased/TaskReleased event for a given task collided on ID across
every lease of that task — unsound for ApplyAdvisory or any future
ID-based lookup. Both now call domain.NewID().
S6: Append's TaskCreated dedup path returned nil (success) without
appending anything. main.go's handler then did
`s.Events(0)[len(s.Events(0))-1]` and returned that — an unrelated event —
with 201 Created, and every other Append caller (Gitea poll/webhook, JSONL
ingest) had no way to distinguish "duplicate, as expected" from "genuinely
appended".
Add domain.ErrDuplicate, returned instead of nil on a duplicate
(source, external_id). Add Store.TaskBySource to resolve the
already-ingested task by that same dedup key. Update every caller:
- main.go's POST /v1/tasks now returns 200 with the existing task on
ErrDuplicate instead of fabricating a 201 with the wrong event.
- provider.Gitea.Poll/IngestWebhook and provider.JSONL.Ingest treat
ErrDuplicate as expected (already-seen issue/line), not a failure —
without this, Gitea polling would have errored out of its loop on the
first already-ingested issue in every batch, since Poll previously
relied on the old nil-on-dup behavior to keep scanning.
TestLeaseAndExpireEventIDsAreUnique and TestTaskBySourceResolvesDuplicate
cover the store-level fixes; TestAppendReplayAndDeduplicate updated for the
new error signal.
AUDIT.md S5, S6.
Router.attempts (checked against RetryPolicy.MaxAttempts) advanced on
every TaskReleased and again on every subsequent lease. Rotation is
TaskReleased carrying a valid handoff_ref (spec §5.3: "rotation =
intra-task lease transfer"), not a failure — so a task healthy enough to
rotate twice hit the default MaxAttempts=3 and was killed by the retry
limit meant for genuine failures (expiry, crash).
HandleEvent now only advances attempts (and applies retry backoff) for a
TaskReleased whose payload lacks a handoff_ref. AssignPending no longer
increments attempts at lease time at all — that was double-counting
against the same failure that a subsequent non-rotation release already
counts.
TestRotationDoesNotCountAgainstRetryLimit drives a task through 5
rotate-and-release cycles with MaxAttempts=3 and asserts it never reaches
TaskFailed.
AUDIT.md B4.
Brief.From/To both serialized as "from" (repeated tag key on one combined
field declaration), and GitSync.Branch/Head/Status all serialized as
"branch" — Go only honors the first `json:"..."` tag on a struct field, so
the second and third tags in `json:"branch" json:"head" json:"status"` were
silently ignored. go vet ./... failed on this; the brief's git state was
unparseable by any client since Head and Status never appeared in the JSON
at all and From/To collided.
Split the combined declarations so each field gets its own tag. go vet
./... now passes clean.
AUDIT.md S1.
System means "the plane itself, in-process" (router, coordinator, adapters,
lease-expiry reclaim) and is unconditionally FullControl with no token
gate. But it was reachable straight from the X-Orchestra-Surface HTTP
header, both in authz.HTTP's token check and in main.go's own `surface`
closure (which every handler actually calls to authorize an event — it
re-parses the header independently of what the HTTP middleware resolved).
Since no deployment configures ORCHESTRA_SYSTEM_TOKEN (no legitimate HTTP
caller should ever need one), tokens[System] is always "", so the token
check was skipped entirely: any LAN request with
"X-Orchestra-Surface: system" got unauthenticated full control to emit any
event on any task.
Both the authz.HTTP middleware and main.go's `surface` closure now
downgrade System to Web before doing anything else with it, so the header
can never resolve to System over HTTP regardless of token config.
AUDIT.md B8.
CLIAdapter.Occupancy called a.Usage(s.PaneID), but ClaudeUsage/CodexUsage/
OpenCodeUsage all take a filesystem path to session state, not a herdr pane
id. Every call failed with "open <pane-id>: no such file", and
Coordinator.rotate silently `continue`d past every failure, so occupancy
always looked unmeasurable and rotation never fired.
Add herdr.Session.SessionFile and CLIAdapter.resolveSessionFile:
- claude: ClaudeSessionFile resolves the transcript by newest-mtime under
Claude Code's own encoded project directory
(~/.claude/projects/<abs-worktree-with-/-as-minus>/*.jsonl). This is the
Phase-1 fallback; the Stop hook's transcript_path (Phase 2) is the
authoritative source once wired.
- codex: routes through the existing CodexActiveUsage sqlite/rollout
discovery instead of the pane id.
- opencode: resolution needs a live session id from the SSE/status API,
not derivable from the worktree alone — refuses loudly with a pointer
to AUDIT.md Phase 1 rather than guessing a path, per the spec's "verify
against a live session before wiring any trigger" (§5.2.1).
A missing/unreadable session file is now a hard error, not a silent
zero-usage Usage{}. SessionHealth gained Occupancy/OccupancyError fields,
populated every refreshSessionHealth tick, so GET /v1/tasks/{id}/health
makes the number rotation decides on observable before trusting it.
Tests: TestClaudeUsageIsLastTurnNotCumulative guards the exact trap named
in §5.2.1 (large early-turn total, small last-turn usage -> low occupancy).
TestClaudeSessionFileNewestByMtime and TestClaudeSessionFileMissingIsHardError
cover the resolver.
AUDIT.md B1. Live verification against a real Claude Code session (the
spec's own acceptance bar for this phase) still needs to happen on a host
with an actual session — not possible from this sandbox.
AdapterFactory.Herdrs is keyed by herdr instance id (e.g. "homesrv-claude"),
but Reconcile, expire, and rotate all looked adapters up by session.Harness
(the harness kind, e.g. "claude"). In production this key never resolves,
so every one of those call sites silently no-ops via a bare `continue`:
orphaned panes are never killed on restart, expired leases never kill their
pane, and rotation exits before it begins.
Add Coordinator.adapterFor(taskID, session), matching the fallback already
used correctly by refreshSessionHealth (HerdrID, then the lease's
HarnessID, then Harness as a last resort), and route all four call sites
through it.
Regression test TestAdapterResolvedByHerdrIDNotHarnessKind registers an
adapter under "homesrv-claude" and leases with Session.Harness == "claude"
(reproducing the real key mismatch) and asserts rotation still fires — the
existing rotation tests used a keyed-by-nothing fake adapter that matched
any lookup string and so masked this bug entirely.
AUDIT.md B2.
Pre-existing uncommitted work found at session start: rotation now emits
anchor_sha on TaskReleased (previously silently dropped by store.Append
validation), multi-repo Gitea provider support, per-project git worktree
roots, and associated test coverage. Committing as a checkpoint before
starting remediation work tracked in AUDIT.md.
kami
kami