14 Commits

Author SHA1 Message Date
kami edf00761fd Tell the planner what it was convened to fix
F66, found live on run 20. A replan reopens the plan phase and the session
it convenes was given the ordinary plan brief and an empty template: no
mention that a plan already exists, what contradicted it, which phase, what
evidence, or that its output supersedes an accepted plan. All of it was
already durable on PlanMismatchRecorded and none of it reached the agent,
so nothing stopped the replacement from sealing with the same
contradiction in it.

The contradiction is now projected onto the task and rendered above the
sealed artifacts, because it changes how they should be read. Its lifetime
is bounded at both ends: a sealed replacement answers it, and so does a
human reply to the stop it caused. Only the phases a mismatch can reopen
render it, since an implementer looks at the code rather than at an account
of the last contradiction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:47:44 +04:00
kami 1330ad9943 Stop reporting a blocker the task is no longer blocked on
F67. TaskCorrected cleared the standing question when a task resumed and
left Blocker and BlockReason in place, so task 29 ran through implement,
review and submission still reporting block_reason plan_mismatch. Every
surface that reads the projection rather than the event log showed a stop
that had already been answered.

The blocker has the same lifetime as the question beside it, and is now
cleared with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:41:36 +04:00
kami cb80dd1952 Record run 20: the mismatch and replan ladder holds, and F66
Both halves proven live. A human_decision mismatch stops the task, the
blocker packet states the contradiction, and a Gitea reply resumes the same
plan in 55 seconds with no reseal. A replan reopens into plan while the old
plan stays accepted with its progress, and sealing the replacement moves
plan_ref, retains the old one in history, stops the old progress counting,
and hands the successor the replacement whole.

F66 recorded: the planning session a replan convenes is told nothing about
the contradiction it exists to fix, though all of it is durable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:39:04 +04:00
kami de18f372d3 Fence the two coordinator-side stops nobody had leased
F65, found live on run 20. A plan mismatch asking for a human decision
recorded its observation, then failed to block the task: Store.Append fences
every lifecycle event on a leased task against the current owner and epoch,
and this TaskBlocked carried neither. The task kept implementing while the
contradiction sat durable in the log, and the agent was told its report was
refused. The trajectory gate had the same omission.

The human-decision path already did this correctly and explained why in a
comment. That comment is now a helper all three call.

The tests could not have caught it. planWith never leased its task, so
every plan test ran in a state no agent can be in, which is exactly what
the lease helper's own comment warns against. It leases now, and the
mismatch block test fails without the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:07:23 +04:00
kami bdc0d4d5be Say that a coordinator deploy leaves the console behind
The ethos console sat undeployed for hours while two coordinator deploys
went out, because both used --no-deps orchestra-api and the web image is
built separately. Checking the commit does not catch it; checking the
served bundle does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:02:29 +04:00
kami 6cb2f932d8 Go and read the reply to a question Orchestra asked
F64. A plan mismatch that asks for a human decision blocks the task, and
nothing came back. Two independent gaps, either one enough to strand it:
the reconciler ran only before a lease and at a turn boundary, so a blocked
task's reply was never even read, and ResumeAnsweredBlockers listed two
block reasons, not this one. PlanMismatchAnswered had no caller anywhere.

Blocked tasks awaiting a reply are now reconciled on their own loop, the
same reasoning the submitted-work loop above it already uses: a task that
cannot be leased cannot be reconciled behind a pre-lease hook. One
predicate, BlockReasonAwaitsReply, now names the set for both loops so they
cannot drift apart again.

The existing test asserted the predicate and never the resume, which is how
this survived. It asserts the resume now, and fails without the fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 19:57:11 +04:00
kami b5c37f693b Bind a manual sign-off to the tree it was given against
F63, found live on run 19. A manual check on these projects is a human
reading what the code prints. RecordPlanPhaseVerification asked only
whether a sign-off for that plan and phase existed, and one exists forever,
so rerunning a phase's automated checks at a new commit carried the human
half along with it. The rig proved it twice: two operator commits and two
re-verification requests, each coming back verified without anyone looking.

The reducer now records which tree the human confirmed, the record carries
it forward as provenance, and a run whose commit does not match it waits
for the human again. A sign-off given before any run has no confirmed tree
and still counts, so the ordinary ordering is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 19:41:50 +04:00
kami bbf3335857 Record run 19: the manual verification seam, and F63
The keyed seam holds. Automated checks alone leave a phase awaiting, a
generic approving comment satisfies nothing, and the keyed form verifies
exactly the phase it names under the plan ref Orchestra supplies. A
re-verification request reruns the plan's own commands at the new tree and
writes a record bound to it.

F63 is what the rig found. A manual sign-off is consulted by existence
alone, so it survives every later tree the human never saw, while the
automated half is re-established at each new SHA. The proposed fix stamps
the satisfied record's sha on the decision and requires it to match.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 19:25:21 +04:00
kami 99b209ba10 Record run 18: an implement successor inherits verified phase progress
The rung the last two runs missed. A five-phase task with ten named checks
kept the implement phase open long enough to rotate inside it. The successor
picked up in implement and its launch context carried the whole sealed plan,
phase-1 and phase-2 as verified, phase-3 as the first unfinished phase, and
the current human authority.

Unasked-for bonus: the progress block renders SHA staleness itself, naming
the tree each phase was verified against and the tree it is now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 17:48:17 +04:00
kami ab3258833d Record run 17: F62 proven live, from a production trigger
The rig was not needed. An implement to review phase change asked for the
handoff through the ordinary path, and the agent ignored it, which is the
exact shape F62 was written for. Every assertion held: a causal release
rather than an idle expiry, four renewals during the bounded wait, one
resend carrying the original reason, the timeout class at 10m5s, no stale
transaction, and a successor that leased normally a minute later.

Also recorded: a five-phase plan does not lengthen the implement phase, and
the state-file lever cannot be driven with systemctl restart alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 17:38:15 +04:00
kami c587f2cc8d Bound the wait for a handoff nobody answers
F62. The rotation is agent-driven: the worker asks, and the agent must write
its handoff. When the agent never does, renewals stopped on the ordinary
progress gate, the lease expired, and the task lost an attempt with nothing on
record saying a handoff had ever been requested. Run 16 showed only "agent
status idle and pane unchanged", 34 times.

The request is now stamped, and the wait around it is bounded. While Orchestra
is explicitly waiting the lease renews, because a quiet pane is the answer the
agent was told to give. The request is re-sent once after four minutes, with
the reason it was first asked with. At ten minutes the worker nacks with
failure class handoff_unanswered, and the coordinator releases the task naming
that cause instead of letting the lease die as generic idleness.

The class is known to DebtClassForFailureClass, so a harness that ignores
handoff requests accumulates as its own debt item rather than hiding inside
lease_expired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 16:21:35 +04:00
kami 41658aea5b Carry the resume order and the tag's meaning in the handoff
The next session should not rebuild either from the commits. A defect
found in experimental work is not a reason to reopen settled
architecture.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 15:51:42 +04:00
kami 765bf2afc6 Hand off with the release path settled and the UI as a truth detector
F57 to F60 close the release-transaction family, F18 closes the single
last_error slot, the debt ledger exists as a read-only projection, and
the operator console is rebuilt on ethos.

The part worth acting on is what the UI could not do honestly: no
web-facing human-decision write path, no keystroke forwarding, context
occupancy trapped in herdr, project configuration unserved, and no
federated handoff request. F61 and F62 are recorded and unbuilt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 14:59:05 +04:00
kami cbd6b11c49 Record run 16: the rotation rung, half proven
A successor inherits the whole sealed plan. Phase progress is withheld
from a review successor on purpose, so the rung still needs an
implement-phase successor. F62: a requested handoff nobody answers is
invisible, and its only consequence is an expiry indistinguishable from
an idle one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 14:14:37 +04:00
49 changed files with 1953 additions and 35 deletions
+420
View File
@@ -2527,3 +2527,423 @@ transaction and no session behind, on either worker.
Both workers hold no sessions and no release transactions. Deployed pair is
still `3c7cf95` on both halves.
## Run 16, 2026-08-29: the rotation rung, half proven
Task `06G4SWEVP71FYYKAV5FV0ZK5ZG` on `34f3c28`, a three-phase plan
(`78d25730140c`) with automated checks inside policy.
### Proven: a successor inherits the whole sealed plan
The launch context of the session that picked the task up after its rotation
carries the complete plan: overview, current state, desired end state,
non-goals, approach, all three phases with their files, changes and
verification, testing strategy, risks, migration, and ten research citations.
### Not a defect: that successor was not given phase progress
`renderPlanProgress` runs only for `WorkPhaseImplement`
(`internal/agentctx/agentctx.go:321`). This successor picked up in review, and
the code says why: an independent review must reconstruct the change from the
diff rather than inherit the implementer's account. Progress is withheld there
deliberately.
So the rung still needs an **implement-phase** successor. The records exist and
are durable, projected by `Task.PlanPhases()`.
### Why the mid-implement rotation missed
A trivial task spends about four minutes in implement, and all three phases
verified inside two and a half of them, every one against the same tree
`3da86dbf8ad5`. The implementer does the whole change first, then verifies each
phase. The suspend, edit and restart that sets `handoff_requested` takes about
a minute, and it landed after the implementer had already asked for review.
Two mechanics worth keeping:
- **The web handoff action cannot drive this.** `RequestHandoff` needs a local
coordinator and every session here is worker-owned, so it answers 503. That
is the Design A guardrail working, and a fourth UI-exposed gap.
- **A state-file edit does not survive a running worker.** It holds sessions in
memory and writes them back. Suspend, edit, then restart; resuming lets the
old copy win. This cost one attempt to learn.
### F62: a requested handoff that is never answered is invisible
The rotation is agent-driven: the worker asks, and the agent must write
`HANDOFF.md` at a turn boundary. The review agent never did. Renewals stopped,
the lease expired at 10:12:05, and the task requeued having lost an attempt.
Worker health recorded exactly one thing, 34 times:
```text
x34 lease 06G4SWEVP71FYYKAV5FV0ZK5ZG not renewed: agent status idle and pane unchanged
```
Nothing says a handoff was requested and left unanswered. There is no timeout,
no retry, and no observation. The only visible consequence is an expiry that
looks identical to an ordinary idle one. Diagnosing it needed the worker state
file.
The ring earned its keep again: one distinct message with a count of 34, rather
than 34 overwrites of one slot.
**Fixed, not yet proven live.** The request is now stamped
(`herdr.Session.HandoffRequestedAt`) and the wait is bounded by
`watchHandoff` in the worker:
- the lease renews while Orchestra is explicitly waiting, because a quiet pane
is the answer the agent was asked for — the renewal gate's new case, bounded
by `handoffAnswerTimeout`;
- the request is re-sent once at `handoffRetryAfter` (4 minutes), with the same
reason it was first asked with;
- at 10 minutes the worker nacks with failure class `handoff_unanswered`, and
the coordinator emits `TaskReleased reason=handoff_unanswered` rather than
letting the lease die as generic idleness.
`DebtClassForFailureClass` knows the class, so a harness that repeatedly
ignores handoff requests now accumulates in the debt ledger instead of hiding
inside `lease_expired`.
Runtime proof still owed: force a request the agent will not answer, and read
the release event rather than the worker state file.
## Run 17, 2026-08-29: F62 proven live on `c587f2c`
Task `06G4V20T528ZTER7KZBVGNAYXC`, a five-phase plan
(`3b16b77054b9`) on the deployed pair, coordinator and worker both at
`c587f2c`.
### The trigger was production, not a rig
The rig planned for this run was a state-file lever. It was never needed. The
`implement -> review` phase change asked for a handoff through the ordinary
path (`rotateForPhase`, reason `phase_changed`), and the agent ignored it
because this task's brief instructed it to from phase 4 onward. Nothing touched
tmux, no process was stopped, and the pane stayed open for the whole wait.
```text
12:48:09 implement -> review, handoff requested, reason=phase_changed
12:50:09 TaskLeaseRenewed agent idle, pane capture unchanged
12:52:15 pane input confirmed the single resend, 4m06s after the request
12:52:39 TaskLeaseRenewed
12:55:14 TaskLeaseRenewed
12:57:49 TaskLeaseRenewed
12:58:14 TaskReleased reason=handoff_unanswered failure_class=handoff_unanswered
last_error="handoff requested (phase_changed) and unanswered for 10m5s"
12:59:14 TaskLeased successor leases normally
13:02:20 TaskSubmitted
```
Every assertion the operator set for this rung held:
- the release does not look like an ordinary `lease_expired`: it carries
`reason=handoff_unanswered` and `lifecycle_phase=handoff_unanswered`;
- the idle-renewal rule did not kill the wait: four renewals on an idle pane
with no capture movement, each one the gate would have refused before this
commit;
- the resend preserved the reason the request was first made with;
- exactly one resend, not one per tick;
- the timeout produced the class at `10m5s`, one tick past the bound;
- nothing stale was left: `releases: {}`, no session, no lease, and zero tmux
panes matching the id;
- the next attempt leased 60s later and reached `TaskSubmitted`.
The debt projection classified it with no new code beyond the class mapping:
```text
v1:operational:handoff_unanswered:workpc-claude:lease operational important
"handoff requested (phase_changed) and unanswered for 10m5s"
```
**Inference, not observation.** The resent prompt's text was not captured. That
it carried the `phase_changed` wording rests on the code path and the timing,
not on a pane capture. A future rig should publish a capture across the resend.
### The implement-successor rung missed again, and why
A five-phase plan does not lengthen the implement phase. This one ran 12:45:18
to 12:48:09, about 2m50s, with verifications 20 to 40 seconds apart:
```text
12:45:47 phase-1 12:46:07 phase-2 12:46:32 phase-3
12:47:12 phase-4 12:47:52 phase-5 12:48:09 implement -> review
```
The brief's explicit one-phase-at-a-time requirement was followed, and it
still took under three minutes. Phase count is not the lever; volume of
mechanical work is.
`awaiting_manual_verification` does not hold the phase open either. All five
phases carried that status and the agent moved on regardless.
### The state-file lever cannot be driven with `restart` alone
The documented procedure is stop, edit, start. The unattended sudo rule grants
`install` and `systemctl restart orchestra-worker`, with no `stop` or `start`.
With `restart` only, the running worker's five-second tick save landed between
the edit and the restart, and the edit was gone. Widening the sudoers line is
the fix.
### One cosmetic note
`Session.HandoffRequestedAt` serializes as `0001-01-01T00:00:00Z` when unset,
because `omitempty` does not omit a zero `time.Time`. The worker parses it back
to a zero value and `watchHandoff` starts the clock, so behaviour is correct.
It is the same marshalling trap the operator console hit.
## Run 18, 2026-08-29: the implement-successor rung, proven
Task `06G4VF5HZW7Q4JBM3TTY7W1Y64` on `c587f2c`, a five-phase plan whose phases
are ten named checks rather than one refactor. Volume of mechanical work is
what keeps the implement phase open; phase count does not.
### The rotation
```text
13:43:10 implement session launched
13:43:44 phase-1 verified
13:44:04 phase-2 verified
13:44:35 handoff requested, reason=milestone, delivered by watchHandoff's resend
13:45:25 TaskLeaseRenewed
13:46:51 TaskReleased handoff_ref 0b4fae6f705a anchor 8ea3c7d1ae8f
13:46:51 TaskLeased successor, same harness
13:46:55 TaskPickupValidated
13:47:06 TaskLaunchAcknowledged, still in implement
```
The lever was the worker state file, backdated past `handoffRetryAfter` so the
request went out on the next tick. Under `c587f2c` that lever now produces a
real prompt: the flag alone used to sit there unasked, which is how run 16
ended in a silent expiry.
### What the successor was given
From `.orchestra/launch.md`, written at 13:47:05:
```text
## Current phase
implement: Implement the accepted plan below. Verify as you go.
## Verified git state
- head: 8ea3c7d1ae8fd960897344694645117adbd8ee82
- uncommitted changes: false
## Current human decisions
None recorded. Work from the goal and acceptance above.
## Plan progress
Orchestra established this by running the plan's own verification. You cannot
write it.
- phase-1 (...): automated checks passed at d3acd4989a20, stale because the
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
- phase-2 (...): automated checks passed at d3acd4989a20, stale because the
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
- phase-3 (Checks 7 to 8): not started
- phase-4 (...): not started
- phase-5 (...): not started
```
Everything the rung asked for is there: the complete accepted plan with all
five phases, their files, changes and verification; the phases already
verified; the first unfinished phase; and the current human authority. The
launch context also carries the accepted research and its dead ends.
**SHA staleness renders itself.** Neither phase is reported as simply passed.
Each says the checks passed at `d3acd4989a20` and are stale because the tree
has moved to `8ea3c7d1ae8f`. That is half of the manual-verification rung
observed without being asked for.
### The lever still loses a race, sometimes
`systemctl restart` alone leaves a five-second window in which the running
worker's tick save can clobber the edit. It clobbered run 17's attempt and
survived run 18's. Stop, edit, start is the reliable sequence, and it needs
`systemctl stop` and `start` in the unattended sudo rule.
## Run 19, 2026-08-29: the manual verification seam, and F63
Two tasks, both ingested from Gitea so that comments reconcile:
`06G4W2TMZ9MW9SEBM5MGTM5288` (issue 24, plan `500464b9`) and
`06G4W63545T3RV8SSKGKN11B3G` (issue 26, plan `53f7918b`).
A burn-in task cannot test this rung. Comments reconcile only for the source
the task came from, so a task created with `source: "burnin"` has no comment
stream at all.
### Proven: the keyed seam
```text
15:09:15 PlanPhaseVerified phase-1 awaiting_manual_verification at 80728967
comment: "Looks good to me. Nice work on this one, ship it."
17:10:03 HumanDecisionRecorded subject=operator_instruction
phase-1 unchanged, still awaiting_manual_verification
comment: "orchestra verify phase-1"
17:10:25 HumanDecisionRecorded subject=plan_phase_verification:500464b9…:phase-1
phase-1 verified
```
All five phases reached `awaiting_manual_verification` on their automated
checks alone. A generic approval satisfied nothing. The keyed form flipped
exactly the phase it named, under the plan ref Orchestra supplied rather than
one the commenter chose.
### Proven: re-verification at a new SHA
On issue 26, inside one live implement lease, with the worker stopped for the
whole edit:
```text
15:23:00 PlanPhaseVerified phase-1 awaiting_manual_verification at f5d8f90b
operator commit moves the tree f5d8f90b -> 1f07b89c
.orchestra/plan-progress.json requests phase-1 again
15:23:17 PlanPhaseVerified phase-1 awaiting_manual_verification at 1f07b89c
17:23:16 HumanDecisionRecorded subject=plan_phase_verification:53f7918b…:phase-1
phase-1 verified at 1f07b89c
```
The rerun ran the plan's own commands at the new tree and wrote a fresh record
bound to it. The worker never reads a command out of the request: the request
names a phase and nothing else about it is trusted.
### F63: a manual sign-off outlives the tree it was given against
The second half of the same rig, with no new comment:
```text
operator commit moves the tree 1f07b89c -> 28af84ff
.orchestra/plan-progress.json requests phase-1 again
15:24:02 PlanPhaseVerified phase-1 verified at 28af84ff
```
`RecordPlanPhaseVerification` consults `manuallySignedOff`
(`internal/operations/planprogress.go:127,152`), which asks only whether a
decision with that subject exists. It does, forever, for that plan ref and
phase id. So the automated half of a phase is re-established at every new SHA
while the manual half is asserted from a tree the human never saw.
This is the same class of bug `AtSHA` exists to prevent, applied to the half
that `AtSHA` does not cover. A manual check on this project is a human reading
printed output; an edit between the sign-off and the rerun can change exactly
that output.
Only a replan clears it, because `PlanPhases()` drops records whose plan ref is
not the accepted one.
Proposed shape, not built: when the reducer flips a record on a keyed decision
(`internal/store/store.go:273`), it knows the record it satisfied and that
record's `AtSHA`. Stamp that sha on the decision, and have `manuallySignedOff`
require it to match the run being recorded. A sign-off that arrives before any
run has no sha and keeps today's behaviour, so the ordinary ordering is
unchanged.
### Also established
- `review -> implement` is legal in `legalPhaseTransitions`, but
`AdvanceWorkPhase` only walks the project's path forward. A task that has
left implement can return only through a review that requests changes, not
through the operator phase endpoint.
- The agent's re-verification request is a file an operator can write:
`.orchestra/plan-progress.json`, `{"phase": "...", "status":
"ready_for_verification"}`. It is the whole lever for this rung.
- Ingest turns every issue label into a required capability, so a rig issue
must carry no labels or it will not schedule.
## Run 20, 2026-08-29: the mismatch and replan ladder, and three defects
Task `06G4WJ9T4F35NZC4Z8QQXM9Z6G` (issue 29) on `de18f37`, plus a first attempt
on `06G4WDJ06G7FK00QF73Z9DDBT4` (issue 27) on `c587f2c` that found F65.
The lever throughout is the agent's own file, written by the operator with the
worker stopped: `.orchestra/plan-mismatch.json`, carrying `phase_id`,
`observed`, `contradicts`, `evidence` and `requested_action`. The worker fills
`plan_ref` and `at_sha` from what it can verify, never from the file.
### F65: a task that stops for a human never stopped
Found on the first attempt, live:
```text
16:04:25 PlanMismatchRecorded phase-2 human_decision
16:04:25 worker: plan mismatch refused: task version conflict
16:04:48 PlanPhaseVerified phase-1 work continues as if nothing happened
```
`Store.Append` fences every lifecycle event on a leased task against the
current owner and epoch. `blockForPlanMismatch` carried neither `harness_id`
nor `lease_epoch`, so the block was refused on every task that was actually
running, which is every task that can produce a mismatch. The observation was
durable and the transition silently did not happen: the repo's own documented
failure shape. `raiseTrajectoryGate` had the identical omission, so the
trajectory gate could never have stopped a live task either.
`RecordPlanMismatch` also returns that error after the record is already
appended, so the worker told the agent its report was refused for a report
Orchestra had accepted.
Fixed in `de18f37` with one `fenceToLease` helper, called by all three stops.
The human-decision path already did it correctly and explained why in a
comment; that comment is now the helper.
**Why no test caught it.** `planWith` never leased its task, so every
plan-progress and plan-mismatch test ran in a state no agent can be in. The
`lease` helper in the same package carries a comment warning about exactly
that. It leases now, and the mismatch block test fails without the fence with
the same `task version conflict`.
### Proven: human_decision stops, and the reply resumes the same plan
```text
16:24:21 PlanMismatchRecorded phase-2 human_decision
16:24:21 TaskBlocked plan_mismatch, lifecycle awaiting_human
18:24:46 HumanDecisionRecorded operator_instruction (Gitea's timestamp)
16:25:16 TaskCorrected resumed
16:25:16 TaskLeased plan_ref unchanged at 0bc8ef84
16:25:21 TaskPickupValidated
```
55 seconds from block to resumed, on the same plan, with no reseal. The
blocker packet states the observation, the plan's claim and the evidence.
This is also the live proof of F64: before `6cb2f93` nothing read a blocked
task's comments, because the reconciler ran only before a lease and at a turn
boundary, and a blocked task is never leased.
### Proven: replan keeps the old plan until a replacement seals
```text
16:26:07 PlanMismatchRecorded phase-3 replan
16:26:07 WorkPhaseChanged plan reopen=PlanMismatchRecorded
during planning: plan_ref 0bc8ef84 accepted, phase-1 and phase-2 counted
after the seal:
plan_ref 922f293b
plan_history [0bc8ef84]
progress empty
successor in implement: four phases, all "not started"
```
The successor's launch context carries the replacement whole, with files,
changes and verification per phase. The old progress stops counting because
`PlanPhases()` drops records whose plan ref is not the accepted one. An
abandoned replan would have cost nothing.
### F66: the planner replans blind
The planning session convened by a replan is given the ordinary plan brief and
an empty template. Its launch context says nothing about the plan that already
exists, the contradiction that reopened the phase, the phase id, the evidence,
or the fact that its output supersedes an accepted plan. The only human input
it carried was an unrelated earlier correction.
Everything it needs is already durable. `PlanMismatchRecorded` holds the phase,
the observation, the contradiction and the evidence, and `reopenPhase` writes
`reopen_phase_id` onto the `WorkPhaseChanged` payload. None of it reaches the
agent, so nothing stops the replacement from sealing with the same
contradiction in it.
### Smaller observation
The implement session kept working after the reopen. It verified phase-2
against the old plan 40 seconds after the task had moved to `plan`. Harmless,
because supersession discards it, but the reopen does not end the session that
reported the contradiction. That session rotates at its own next boundary.
+8
View File
@@ -66,6 +66,14 @@ the live herdr instance and check.
`docker logs orchestra-api`. **Deploying a code change means rebuilding the
compose images** (`up -d --build`) — the running image can silently predate
recent commits, so compare its build time against `git log`.
- **A coordinator deploy does not deploy the console.** They are two images
built from the same repo, and the usual `up -d --no-deps orchestra-api`
leaves `orchestra-web-ui` on whatever it was. This has bitten twice: the
ethos console landed in the repo on 2026-08-29 02:44 and was still serving a
2026-07-30 image hours later, through two coordinator deploys. Rebuild it
explicitly from the same clean worktree (`docker build` in `web/`, then
`up -d --no-deps orchestra-web-ui`), and check the served bundle rather than
the commit: `curl -s http://127.0.0.1:19145/assets/<css> | grep 8F7AE5`.
- `orchestra.service` was the **previous** deployment; the unit file was
deleted from `deploy/` on 2026-07-31 along with `redeploy.sh` (which
`sudo install`ed to `/usr/local/bin` and restarted it). If a stale copy is
+349
View File
@@ -0,0 +1,349 @@
# Handoff: the release path is settled, and the UI became a truth detector
Written 2026-08-29, 14:20 local (10:20 UTC). Read with `BURNIN.md` (the run
ledger, current through run 16), `DEBT-DESIGN.md`, `PLAN-SPEC-DESIGN.md`,
`AUDIT.md` and `CLAUDE.md`. The previous handoff is
`HANDOFF-2026-08-28-plan-v1.md`.
Everything below was observed live unless it says otherwise.
## The headline
**The expired-release defect is fixed and proven both ways.** The whole family
around it is closed. F57 through F60 settle what happens to a release
transaction in every case. That includes the ones that used to need an operator
with a text editor.
**Three new things exist that did not before.** A bounded observation ring on
worker health, closing F18. A read-only debt ledger projected from the event
log. An operator console rebuilt on the ethos design system.
**The UI turned out to be a truth detector.** Nine screens were built against
real endpoints. They found four places where Orchestra has no capability to
support the intended interface. That list is the most valuable output of the
session.
```text
orchestra-plan-v1 plan machinery proven
orchestra-f18-baseline d6ee10f, the bounded observation ring
↓ 6 commits
34f3c28 deployed now: release path settled, debt ledger, new UI
```
## Deployed state
| Half | Revision |
|---|---|
| Coordinator, homesrv container | `34f3c28` |
| Worker, workpc systemd | `34f3c28` |
```text
commit 34f3c2888fc7d45d190d93e1ea42501b6cd3e474
coordinator sha256 1d32d83ec859b36e12473fab01bbfc3c769b97b4f6ae4db2841db5161e0eda19
worker sha256 0e3877321dea8a1eeda51ccc6f3ead1a14aa5f5cae4d95b704f61c364248ec65
```
`cbd6b11` is one commit above and is documentation only. Three commits are
unpushed.
**Worker installs no longer need a human.** The operator installed the
`/etc/sudoers.d` line, so `sudo -n install …` and
`sudo -n systemctl restart orchestra-worker` both work unattended. Verify the
running revision from the journal, never the installed file.
## The defects fixed, and how each was found
Not one came from reading code. Every one came from a live run failing.
| Id | Commit | What |
|---|---|---|
| F57 | `6565b9f` | An expired lease could never commit the anchor it had already pushed. The worker sent an epoch the expiry replay had deleted, and the coordinator refused any handoff without a live lease. The epoch now belongs to the transaction, `TaskReleased` retains the ending epoch, and `lateHandoffAccepted` lets exactly that owner commit while the task is queued and unleased. |
| F58 | `03663f4` | A superseded transaction retried a permanent 409 forever, holding the pane and pinning `ActiveTask`. Run 10's task did it for seven hours. `TaskLeased` now abandons a transaction whose id the lease does not carry. |
| F59 | `8e37989` | F58 fires on `TaskLeased`, and a failed task is never leased again. `TaskFailed` now drops the transaction too. |
| F60 | `3c7cf95` | The general rule the other two were reaching for. Terminal is failed or completed. Blocked keeps the transaction, because a reopen can still commit it, so `TaskBlocked` now retains the ending epoch as well. A refusal parks the commit for 30s backing off to 5 minutes, and any event about the task un-parks it. A transport failure is not an answer and retries at once. |
| F18 | `d6ee10f` | The single `last_error` slot. Worker health now carries up to sixteen distinct observations with repeat counts and first/last times, collapsing by message rather than by position. |
### The rig that proved F57, and the guard
```text
19:00:10.742 transaction opens at prepared, anchor pushing
19:00:10.727 TaskReleased v15 reason=lease_expired surface=tui
19:00:11.662 TaskReleased v16 the late commit, accepted 935ms after the lease died
19:00:11.665 TaskLeased v17 successor picks up the handoff
19:00:15.524 TaskPickupValidated v18
```
Race guard, next boundary: force the expiry, then lease the task to a probe
harness before the push finishes. The late commit is refused, no handoff is
written, and the successor's lease stands.
**The rig technique matters more than the rig.** Suspending the worker cannot
produce this ordering. The event replay runs at the top of every tick and
discards the transaction. The ordering exists only inside one call:
transaction opened, anchor pushing, commit not yet sent. So poll the worker
state file at 2ms and fire `POST /v1/tasks/<id>/release` the instant a
transaction appears at `prepared`.
**A named probe harness owns a lease without starting an agent.**
`race-guard-probe` never picks anything up and expires on the normal TTL.
## Corrections to the previous handoff
**The operator lifecycle actions do not lose a version race.** On a leased
task, `block`, `release` and `attention` are refused by
`internal/store/store.go:885-901` when the payload omits `harness_id` and
`lease_epoch`. Ten attempts in 550ms all failed that way. Send both fencing
fields and they succeed on the first try.
**The OpenCode Zen free tier is not blocked.** The selected model was.
## The debt ledger
`DEBT-DESIGN.md` answers nine design questions and carries four amendments the
operator made. Slice one is built, deployed and run.
**Slice one writes nothing.** A read-only projection over the existing log,
plus a pure eligibility function and `GET /v1/debt`. It folded 881 events and
produced 15 candidates and 3 gaps.
```text
v1:operational:lease_expired:workpc-opencode:lease r=41 tasks=4
v1:operational:lease_expired:workpc-claude:lease r=29 tasks=13
v1:operational:lease_failure:-:lease r=20 tasks=16
v1:correctness:handoff_validation:-:lease r=5 tasks=5
```
The opencode failure shape is the top item, found mechanically. Run 14 reached
the same conclusion by hand from a pane capture.
**It reported what it cannot see, which was the point.** The 409 release loop
does not appear. That evidence lived in the F18 worker ring, and no event
carries it. Manual interventions are a non-durable gap for the same reason.
**The first run exposed four defects in the model**, all recorded at the end of
`DEBT-DESIGN.md`:
- the component part is too coarse for lease evidence
- `harness` is often empty on block reasons
- path normalization mangled a mismatch reference
- recurrence alone is the wrong sort order
Layout, so the read model does not end up in the command layer:
```text
internal/domain/debt.go types, signature, classification
internal/store/debt_projection.go the fold, and the gap report
internal/operations/debt.go CheckDebtEligibility
```
## The operator console
Nine screens on the ethos system, signal violet `#8F7AE5`, routing fork motif.
Each screen was built by its own agent against a foundation with one author.
The shell, tokens and primitives could not drift into nine dialects.
**Render before signing off.** Three bugs existed that no computed value would
have caught. All three came from looking at a screenshot:
- The previous stylesheet fought every shared class name and leaked properties
the new rules never mention, which is how `position: fixed` survived on
`.topbar`. It is now scoped under `.legacy` and reaches only the login route.
That also stops its green accent and its `backdrop-filter` from reaching the
console.
- Go marshals a zero `time.Time` as `0001-01-01T00:00:00Z` and `omitempty` does
not omit a struct, so absent timestamps arrived populated-looking and
rendered as `739855d ago`. Stripped once in `client.ts`, with a test.
- Long machine ids overflowed their cards and painted under the next one.
Chromium is installed at `/usr/bin/chromium`. To see a screen without a live
session, write a throwaway harness that stubs `window.fetch` and renders
`<Console>` inside a `MemoryRouter`, served by vite on a spare port. Note that
`npx` and `./node_modules/.bin/*` do not work on this filesystem: call
`node ./node_modules/vite/bin/vite.js` directly.
## What the UI proved Orchestra cannot do
This is the part worth acting on. Each screen refused to fake something, and
the refusals name real capability gaps.
| Gap | Evidence |
|---|---|
| **No web-facing human-decision write path** | `Steer / Correct` is disabled. `internal/ui/ui.go`'s action switch has grant/deny approval, resubmit, handoff, release, block and complete, and nothing writes `HumanDecisionRecorded`. The spec makes steering the primary action of the task detail screen. |
| **No keystroke forwarding** | `Take control` is disabled. Only resubmit and approval grant/deny reach a live pane. |
| **Context occupancy is trapped in herdr** | Three screens independently hit it. No projection carries it. |
| **Project configuration is not served** | Repo, remote, quality gate and verification policy live only in `config.jsonc`. The projects screen can show none of it. |
| **The web cannot request a handoff for a federated task** | `RequestHandoff` needs a local coordinator and answers 503. That is the Design A guardrail working. |
The operator's direction on these. Treat first-class direct human input as the
highest-value backend feature. Build it as `POST /v1/tasks/<id>/decisions`,
using the same durable decision semantics as Gitea comments, so Gitea, CLI and
web converge on one `HumanDecisionRecorded`. Keep take-control disabled,
because arbitrary pane input bypasses the durable authority model. Expose
occupancy through a session health projection rather than teaching the web
server about herdr. Add a read-only effective project configuration endpoint,
which F61 will also need.
## Where the plan-machinery ladder stands
Proven in run 14: the worker executes the **sealed plan's** commands rather
than the request's, and every `PlanPhaseVerified` binds `plan_ref`, `phase_id`,
`at_sha`, `evidence_ref`, `lease_epoch` and `harness_id`.
Proven in run 16: a successor inherits the **whole sealed plan**, all phases
with their files, changes, verification, and the research citations.
**Still unproven, and the next runtime item:**
```text
mid-implement rotation
→ successor picks up in implement
→ launch context states which phases are already verified
manual verification
→ SHA goes stale
→ re-verification
plan mismatch
→ human decision
→ real replan, old plan retained, replacement launched
```
Two things make the first one hard, and both are now known:
- **A trivial task spends about four minutes in implement**, and verifies every
phase against one tree near the end. The implementer writes the whole change
first, then verifies each phase in turn. Use a task whose implement phase
genuinely runs long.
- **A state-file edit does not survive a running worker.** It holds sessions in
memory and writes them back. Suspend, edit, then `sudo systemctl restart`.
Resuming lets the old copy win. Setting `handoff_requested` on the session is
the production rotation lever.
Progress is withheld from a **review** successor on purpose
(`internal/agentctx/agentctx.go:321`), because an independent review must
reconstruct the change from the diff. That absence is not the defect.
## F61 and F62, recorded and not built
**F61: the planner learns the verification policy by refusal.** The brief says
a policy exists, not what is in it. Every plan therefore pays one refused round
trip. The recovery loop works: run 14's planner consumed the refusal and
resealed 31 seconds later. It is not a one-liner, because `agentctx.Input.Policy` is filled
from the worker's `SafeOperations` while the verification policy is
coordinator-side. The reason to promote it later is local models, which may
propose forbidden commands repeatedly because they cannot infer the allowed
substitute.
**F62: a requested handoff nobody answers is invisible.** The rotation is
agent-driven. In run 16 the agent never wrote `HANDOFF.md`, renewals stopped,
the lease expired, and the task lost an attempt. Worker health recorded only
`agent status idle and pane unchanged`, 34 times. There is no timeout, no
retry, and no observation saying a handoff was requested and left unanswered.
The expiry is indistinguishable from an ordinary idle one.
## Live state
Three pull requests are open and unreviewed:
```text
06G4M6HF1Z3EREX1X3NEKSHP24 pulls/18
06G4M8WHGQ4P3GQMPEEH0RJRHM pulls/19
06G4SWEVP71FYYKAV5FV0ZK5ZG pulls/20
```
Both workers are online on `34f3c28` with no release transactions and no
sessions. 29 blocked `test-e2e` tasks are burn-in debris. Three `correx` tasks
are queued and unschedulable, because `correx` has no entry in the
coordinator's `config.jsonc`.
**opencode now runs.** The model was the problem, and `hy3-free` works. It then
stops on a permission prompt, because `~/.config/opencode/opencode.jsonc` sets
`"bash": "ask"`. Orchestra can answer that exact dialog, but only when an
operator queues `grant_approval`, so unattended work stalls on the first
command. Set `"bash": "allow"` for unattended runs. That file also has no
top-level `model` key, so OpenCode picks whatever sits at the top of
`~/.local/state/opencode/model.json`, which any manual pick silently changes.
**The opencode adapter is now a debt item, not a curiosity.** It cannot resolve
a session file. Activity therefore reads `unknown`, and the worker cannot tell
finished from never-started. The debt projection surfaced it independently as
the top recurring operational item.
## Things that will bite
- **Background python tasks get killed here.** Three watchers died before doing
anything. Foreground polling and the `Monitor` tool both work.
- **`/v1/events` is one line of JSON.** A `grep` for two substrings matches
across unrelated tasks. Parse it.
- **`npx` and `./node_modules/.bin/*` fail on this filesystem.** Call node
directly.
- **`rm` and `cp` are interactive.** Use `/bin/rm -f` and `install`.
- **Secrets are guarded.** Expand a token inside the container in one remote
command:
`T=$(docker exec orchestra-api printenv ORCHESTRA_TUI_TOKEN); curl -s -H "X-Orchestra-Surface: tui" -H "Authorization: Bearer $T" ...`
- **Rebuild the coordinator from a detached worktree**, and split the worktree
add, the docker build and the compose up into separate commands.
- **`deploy/build.sh` builds both halves from one commit with one stamp.** Use
it.
- **Geist is not on disk.** Both stacks fall back to system faces, and the
ethos threat model rules out the font CDN.
## Resume in this order
Set by the operator at the session boundary. Do not rebuild it from the commits.
1. **F62 first.** Make a requested-but-unanswered handoff visible and bounded.
Preserve the lease while Orchestra is explicitly waiting, retry the confirmed
request, then emit a causal timeout. Today it ends as generic idleness.
2. **Repeat the implement-successor rung** with a deliberately longer task. Get
at least one `PlanPhaseVerified`, force the handoff while still in
`implement`, and prove the successor sees the complete accepted plan, the
verified previous phase, the first unfinished phase, and the current human
authority.
3. **Finish the remaining ladder.** Manual verification, SHA staleness and
reverification, human-decision mismatch, then a real replan with old-plan
provenance.
4. **Use the capability table above as backend work discovery.** Do not add fake
controls. Each disabled action is concrete evidence of a missing capability.
5. **Continue the debt slices independently.** Durable worker observations and
manual-intervention events are the next evidence gaps. Not automatic
maintenance yet.
## What the tag means
```text
orchestra-release-v1 -> 34f3c28
deployed, settled lifecycle plus the truth-detector UI baseline
later HEADs
experimental plan, debt and runtime work that must earn their own
release proof
```
Use that distinction. A defect found in experimental work is not a reason to
reopen settled architecture.
## The roadmap the operator set
```text
A. runtime correctness finish the plan-machinery live proof
B. evidence and debt durable worker observations, manual-intervention
events, then rerun the projection
C. operator surface first-class HumanDecision write API,
effective project-config read API,
session and context health projection
D. adapter fix opencode activity and session resolution
E. UI wire capabilities as backend support becomes real
```
The framing that ties them together is the thing to keep. The UI says what
Orchestra cannot expose or control. The debt ledger says which of those
shortcomings repeatedly costs something. The burn-in says which runtime
semantics are reliable. Those three decide what gets built next.
One design question to settle before B's slice two writes any code. The worker
ring is bounded and lossy by construction. Ingesting it durably means deciding
whether the coordinator stores every observation as an event, or only
transitions. Storing every heartbeat's ring would write the same 41-count
observation hundreds of times.
+17
View File
@@ -0,0 +1,17 @@
# orchestra ui mockups
`final/` contains the nine accepted/current screen directions:
1. dashboard
2. task detail
3. terminal live pane
4. tasks
5. decisions
6. workers
7. projects
8. review
9. settings
`iterations/` contains every generated mockup from the design session, including superseded variants.
`orchestra-ui-spec.md` is the accompanying implementation/design specification.
+110
View File
@@ -0,0 +1,110 @@
package main
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"orchestra/internal/domain"
"orchestra/internal/federation"
"orchestra/internal/herdr"
)
// F62. Run 16: the agent was asked to hand off, never wrote HANDOFF.md,
// renewals stopped, and the lease died as ordinary idleness. Waiting is now
// bounded: re-ask once, then give the task up with a class that says why.
func TestUnansweredHandoffIsRetriedThenGivenUp(t *testing.T) {
var nack map[string]any
w, backend, _, done := phaseWorker(t, func(rw http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/nack") {
_ = json.NewDecoder(r.Body).Decode(&nack)
}
rw.Write([]byte(`{}`))
})
defer done()
ctx := context.Background()
requested := func(ago time.Duration) herdr.Session {
s := w.sessions["task"]
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
s.HandoffRequestedAt = time.Now().UTC().Add(-ago)
w.sessions["task"] = s
return s
}
// Still inside the answering window: nothing said, nothing given up.
if s, gaveUp := w.watchHandoff(ctx, "task", requested(time.Minute)); gaveUp || s.HandoffRetried {
t.Fatalf("gave up while still waiting: gaveUp=%v session=%+v", gaveUp, s)
}
if len(backend.prompts) != 0 {
t.Fatalf("re-asked too early: %q", backend.prompts)
}
// Past the retry point: asked again, exactly once.
s, gaveUp := w.watchHandoff(ctx, "task", requested(handoffRetryAfter+time.Minute))
if gaveUp || !s.HandoffRetried || len(backend.prompts) != 1 {
t.Fatalf("retry: gaveUp=%v retried=%v prompts=%q", gaveUp, s.HandoffRetried, backend.prompts)
}
if _, gaveUp = w.watchHandoff(ctx, "task", s); gaveUp || len(backend.prompts) != 1 {
t.Fatalf("re-asked every tick: %q", backend.prompts)
}
// Past the bound: a causal reclaim, and no lease left to renew.
s = requested(handoffAnswerTimeout + time.Second)
s.HandoffRetried = true
w.sessions["task"] = s
if _, gaveUp = w.watchHandoff(ctx, "task", s); !gaveUp {
t.Fatal("an unanswered handoff waited forever")
}
if nack["failure_class"] != "handoff_unanswered" {
t.Fatalf("nack = %+v", nack)
}
if detail, _ := nack["last_error"].(string); !strings.Contains(detail, "phase_changed") {
t.Fatalf("the reclaim does not name the request: %q", detail)
}
if _, held := w.leases["task"]; held {
t.Fatal("the given-up task kept its lease")
}
}
// The lease must survive the wait it was asked to make: an idle pane is the
// answer Orchestra requested, not evidence of an agent that stopped working.
func TestWaitingForAHandoffKeepsTheLease(t *testing.T) {
renewals := 0
api := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
renewals++
rw.Write([]byte(`{}`))
}))
defer api.Close()
backend := &recordingBackend{status: "idle", progress: "same screen"}
w := &worker{
api: federation.Client{BaseURL: api.URL, WorkerID: "h", Token: "t"},
backend: backend,
harness: "claude",
sessions: map[string]herdr.Session{"task": {PaneID: "pane", HandoffRequested: true, HandoffRequestedAt: time.Now().UTC()}},
leases: map[string]lease{"task": {Epoch: "e", Version: 1, Until: time.Now(), ProgressSHA: domain.Hash([]byte("same screen"))}},
quarantined: map[string]bool{},
statePath: filepath.Join(t.TempDir(), "state.json"),
}
w.renewLeases(context.Background())
if renewals != 1 {
t.Fatalf("a lease waiting on a requested handoff renewed %d times, want 1", renewals)
}
// Past the bound the exemption stops: watchHandoff has given the task up
// by then, and nothing keeps an unanswered request alive.
s := w.sessions["task"]
s.HandoffRequestedAt = time.Now().UTC().Add(-handoffAnswerTimeout - time.Second)
w.sessions["task"] = s
l := w.leases["task"]
l.Until = time.Now()
w.leases["task"] = l
w.renewLeases(context.Background())
if renewals != 1 {
t.Fatalf("the exemption outlived its bound: renewals=%d", renewals)
}
}
+80 -3
View File
@@ -174,6 +174,7 @@ func releaseBackoff(attempts int) time.Duration {
}
return d
}
type projectConfig struct {
Repo string `json:"repo"`
Root string `json:"worktree_root"`
@@ -673,6 +674,13 @@ func (w *worker) releaseReady(ctx context.Context) {
w.advanceRelease(ctx, id, s)
continue
}
if s.HandoffRequested {
next, gaveUp := w.watchHandoff(ctx, id, s)
if gaveUp {
continue
}
s = next
}
w.rotationTick(ctx, id, s)
}
}
@@ -772,7 +780,7 @@ func (w *worker) rotationTick(ctx context.Context, id string, s herdr.Session) {
w.recordError(fmt.Errorf("rotation %s threshold prompt: %w", id, err))
return
}
s.HandoffRequested, s.HandoffReason = true, d.Reason
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, d.Reason, time.Now().UTC()
w.sessions[id] = s
_ = w.save()
}
@@ -1166,6 +1174,70 @@ func (w *worker) submit(ctx context.Context, id string, s herdr.Session, e compl
// the directory also holds for a worktree that has no inner .gitignore.
const stageExclude = ":!.orchestra"
// F62: a requested handoff nobody answers was invisible. Renewals stopped,
// the lease expired, and the task lost an attempt with nothing on record
// saying a handoff had ever been asked for — worker health showed only "agent
// status idle and pane unchanged", 34 times in run 16.
const (
handoffRetryAfter = 4 * time.Minute
handoffAnswerTimeout = 10 * time.Minute
)
// watchHandoff bounds the wait for an agent's handoff answer: re-send the
// request once, then give the task up with a class that names the cause. It
// returns the session to keep using and whether the task was given up.
func (w *worker) watchHandoff(ctx context.Context, id string, s herdr.Session) (herdr.Session, bool) {
if s.HandoffRequestedAt.IsZero() {
// A session persisted before the stamp existed, or requested by a path
// that does not set it. Start the clock now rather than time out a
// request retroactively.
s.HandoffRequestedAt = time.Now().UTC()
w.sessions[id] = s
_ = w.save()
return s, false
}
waited := time.Since(s.HandoffRequestedAt)
if waited < handoffRetryAfter {
return s, false
}
if waited < handoffAnswerTimeout {
if s.HandoffRetried || w.executionBackend() == nil {
return s, false
}
a := herdr.CLIAdapter{Backend: w.executionBackend(), Harness: w.harness}
var err error
if s.HandoffReason != "" {
err = a.RequestHandoffReason(ctx, s, s.HandoffReason, nil)
} else {
err = a.RequestHandoff(ctx, s)
}
if err != nil {
w.recordError(fmt.Errorf("handoff %s re-request: %w", id, err))
return s, false
}
s.HandoffRetried = true
w.sessions[id] = s
_ = w.save()
w.recordError(fmt.Errorf("handoff %s (%s) unanswered for %s: request re-sent", id, s.HandoffReason, waited.Round(time.Second)))
return s, false
}
l, ok := w.leases[id]
if !ok {
return s, false
}
detail := fmt.Sprintf("handoff requested (%s) and unanswered for %s", s.HandoffReason, waited.Round(time.Second))
if err := w.api.NackStart(ctx, id, l.Epoch, l.Version, "handoff_unanswered", detail, w.sessionEvidence(ctx, id, s)); err != nil {
w.recordError(fmt.Errorf("handoff timeout %s: %w", id, err))
return s, false
}
// The coordinator answers with TaskReleased; its replay quarantines the
// pane. Drop the lease here so nothing renews it in the meantime.
delete(w.leases, id)
_ = w.save()
w.recordError(errors.New(detail))
return s, true
}
func (w *worker) renewLeases(ctx context.Context) {
if w.executionBackend() == nil {
return
@@ -1199,6 +1271,11 @@ func (w *worker) renewLeases(ctx context.Context) {
case l.ProgressSHA == "":
// First renewal has no baseline to compare against. Record one and
// allow this renewal; the next one must show real movement.
case s.HandoffRequested && !s.HandoffRequestedAt.IsZero() && time.Since(s.HandoffRequestedAt) < handoffAnswerTimeout:
// Orchestra told this agent to stop and write its handoff. A quiet
// pane is the answer it was asked for, so the lease is held while
// the wait is explicitly bounded (F62). Past the bound the case
// stops matching and watchHandoff has already given the task up.
default:
w.recordError(fmt.Errorf("lease %s not renewed: agent status %s and pane unchanged since the last renewal", taskID, status))
continue
@@ -1932,7 +2009,7 @@ func (w *worker) federatedTurn(ctx context.Context, id string, a herdr.Adapter,
w.recordError(fmt.Errorf("reconcile failure handoff %s: %w", id, err))
return
}
s.HandoffRequested, s.HandoffReason = true, "reconcile_failure"
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "reconcile_failure", time.Now().UTC()
w.sessions[id] = s
_ = w.save()
return
@@ -2009,7 +2086,7 @@ func (w *worker) rotateForPhase(ctx context.Context, id string, a herdr.Adapter,
w.recordError(fmt.Errorf("phase rotation %s: %w", id, err))
return
}
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "phase_changed", time.Now().UTC()
w.sessions[id] = s
_ = w.save()
log.Printf("phase changed for %s: session rotating", id)
+25
View File
@@ -1589,6 +1589,12 @@ func main() {
case "invalid_handoff":
typ = "TaskBlocked"
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonHandoffValidation), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_nacked", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
case "handoff_unanswered":
// F62: not a launch failure. The agent was asked to hand off
// and never did, so the reclaim says exactly that instead of
// arriving as an ordinary idle expiry.
typ = "TaskReleased"
p, _ = json.Marshal(map[string]any{"reason": "handoff_unanswered", "failure_class": b.FailureClass, "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "handoff_unanswered", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
case "launch_uncertain":
typ = "TaskNeedsAttention"
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonLeaseFailure), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_uncertain", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
@@ -1825,6 +1831,25 @@ func main() {
coordinator.ReconcileFailureHandoff = v
}
}
// The third reconciliation point, and the same reasoning the submitted
// work loop above already uses: a blocked task cannot be leased, so a
// pre-lease hook can never see the reply that should unblock it (F64).
// Orchestra stopped and asked the human a question; nothing else here
// will go and read the answer.
go func() {
ticker := time.NewTicker(time.Minute)
defer ticker.Stop()
for range ticker.C {
for _, t := range s.Tasks() {
if t.State != domain.StateBlocked || !domain.BlockReasonAwaitsReply(t.BlockReason) {
continue
}
if err := reconciler.Reconcile(context.Background(), t.ID); err != nil {
log.Printf("reconcile blocked task %s: %v", t.ID, err)
}
}
}
}()
}
if path := os.Getenv("ORCHESTRA_JSONL"); path != "" {
sup := &provider.Supervisor{Name: "jsonl", Run: func(ctx context.Context) error {
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

+32
View File
@@ -314,6 +314,9 @@ func renderTask(in Input) string {
fmt.Fprintf(&b, "- head: %s\n", fallback(in.Git.HeadSHA))
fmt.Fprintf(&b, "- uncommitted changes: %t\n", in.Git.Dirty)
// Above the sealed artifacts, because it is the reason this session exists
// and it changes how the artifacts below should be read.
b.WriteString(renderReopen(in))
b.WriteString(renderSealed(in))
// Below the plan, above continuity. Progress is a fact about the plan, so
// it follows the plan; continuity is one predecessor's account, so it
@@ -499,6 +502,35 @@ func renderPlanProgress(in Input) string {
return b.String()
}
// renderReopen states the contradiction that reopened this phase. Without it
// the planning session a replan convenes gets the ordinary plan brief and an
// empty template, with nothing saying a plan already exists, what contradicted
// it, or that its output supersedes an accepted plan (F66). Everything here is
// durable on PlanMismatchRecorded; it simply never reached the agent.
//
// Only the phases a mismatch can reopen render it. An implementer that finds
// the next contradiction is looking at the code, not at this account of the
// last one.
func renderReopen(in Input) string {
m := in.Task.PlanMismatch
if m == nil || (in.Phase != domain.WorkPhasePlan && in.Phase != domain.WorkPhaseResearch) {
return ""
}
var b strings.Builder
b.WriteString("\n## Why this phase reopened\n\n")
fmt.Fprintf(&b, "A plan was already accepted and the code contradicted it. Orchestra reopened this phase to settle that, and the session that found it is gone.\n\n")
fmt.Fprintf(&b, "- phase: %s\n", collapse(m.PhaseID))
fmt.Fprintf(&b, "- observed: %s\n", collapse(m.Observed))
fmt.Fprintf(&b, "- the plan says: %s\n", collapse(m.Contradicts))
for _, e := range m.Evidence {
fmt.Fprintf(&b, "- evidence: %s\n", collapse(e))
}
if in.Phase == domain.WorkPhasePlan {
b.WriteString("\nThe accepted plan stays accepted until you seal a replacement, and sealing one supersedes it along with every phase it had verified. Address the contradiction above: a replacement that repeats it will be contradicted again.\n")
}
return b.String()
}
func renderSealed(in Input) string {
var b strings.Builder
research := in.Research
+58
View File
@@ -734,3 +734,61 @@ func TestPendingManualGateRendersStale(t *testing.T) {
t.Errorf("a pending manual gate at HEAD rendered stale:\n%s", fresh)
}
}
// F66, found live on run 20. A replan convenes a planning session to settle a
// contradiction, and that session was told nothing about it: the ordinary plan
// brief, an empty template, and no mention that a plan already exists.
func TestAReopenedPlanPhaseIsToldWhatContradictedThePlan(t *testing.T) {
in := input()
in.Phase = domain.WorkPhasePlan
in.Task.PlanRef = "plan-a"
in.Task.PlanMismatch = &domain.PlanMismatch{
PlanRef: "plan-a", PhaseID: "phase-3", AtSHA: "18ccaf00000000000000000000000000000000aa",
Observed: "the aggregation runs per figure, not per person",
Contradicts: "the plan states the pipeline already aggregates per person",
Evidence: []string{"internal/figures/aggregate.go:88"},
RequestedAction: domain.PlanMismatchReplan,
}
out, err := Build(in)
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
"## Why this phase reopened",
"phase: phase-3",
"the aggregation runs per figure",
"the plan states the pipeline already aggregates per person",
"internal/figures/aggregate.go:88",
"stays accepted until you seal a replacement",
} {
if !strings.Contains(out.Task, want) {
t.Fatalf("the planner was not told %q:\n%s", want, out.Task)
}
}
// It must come before the artifacts it changes the reading of.
in.Research = &workphase.Research{Findings: []workphase.Finding{{
ID: "r1", Confidence: "fact", Claim: "aggregation is per figure",
Evidence: "internal/figures/aggregate.go:88",
}}}
out, err = Build(in)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(out.Task, "## Accepted research") {
t.Fatal("the research this planner works from is missing")
}
if strings.Index(out.Task, "## Why this phase reopened") > strings.Index(out.Task, "## Accepted research") {
t.Fatal("the reason this session exists is rendered below the material it qualifies")
}
// An implementer is looking at the code, not at an account of the last
// contradiction, so it renders nowhere else.
in.Phase = domain.WorkPhaseImplement
out, err = Build(in)
if err != nil {
t.Fatal(err)
}
if strings.Contains(out.Task, "## Why this phase reopened") {
t.Fatal("a settled contradiction reached the implementation phase")
}
}
+1 -1
View File
@@ -251,7 +251,7 @@ func DebtClassForBlockReason(r BlockReason) (DebtClass, bool) {
// classes a worker actually emits are listed; an unknown one is not guessed at.
func DebtClassForFailureClass(f string) (DebtClass, bool) {
switch f {
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired":
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired", "handoff_unanswered":
return DebtOperational, true
case "invalid_handoff":
return DebtCorrectness, true
+33 -9
View File
@@ -82,6 +82,23 @@ const (
BlockReasonUnknown BlockReason = "unknown"
)
// BlockReasonAwaitsReply reports whether this stop is a question waiting on a
// human, rather than a fault or a budget. Every reason listed here resumes on
// a reply, so both the loop that goes and reads replies and the loop that
// returns answered tasks to the queue must agree on the set. They did not:
// plan_mismatch blocked for an answer that nothing ever went to fetch, and
// nothing would have resumed it if it had (F64).
//
// operator_required is deliberately absent. A task that has spent its question
// budget continues when an operator says so, not when someone replies.
func BlockReasonAwaitsReply(r BlockReason) bool {
switch r {
case BlockReasonHumanDecision, BlockReasonTrajectoryGate, BlockReasonPlanMismatch:
return true
}
return false
}
func (r BlockReason) Valid() bool {
switch r {
case BlockReasonLeaseFailure, BlockReasonWorkerOffline, BlockReasonLeaseExpired,
@@ -168,19 +185,19 @@ type Task struct {
QualityGate string `json:"quality_gate,omitempty"`
// Block evidence is projected from TaskBlocked so terminal records remain
// diagnosable after the live coordinator mapping is gone.
Blocker string `json:"blocker,omitempty"`
BlockReason BlockReason `json:"block_reason,omitempty"`
BlockedAt time.Time `json:"blocked_at,omitempty"`
LastPaneID string `json:"last_pane_id,omitempty"`
LastHarness string `json:"last_harness_id,omitempty"`
Blocker string `json:"blocker,omitempty"`
BlockReason BlockReason `json:"block_reason,omitempty"`
BlockedAt time.Time `json:"blocked_at,omitempty"`
LastPaneID string `json:"last_pane_id,omitempty"`
LastHarness string `json:"last_harness_id,omitempty"`
// LastLeaseEpoch is the fencing token of the lease that most recently
// ended. A worker can push its release anchor and only then discover the
// lease expired; the finished work is durable in git but the commit can
// never land. Retaining the epoch lets exactly that owner still commit
// while the task sits unleased.
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
LastSession SessionEvidence `json:"last_session,omitempty"`
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
LastSession SessionEvidence `json:"last_session,omitempty"`
// Recovery state is part of the durable projection, never process-local
// router memory. This makes retry and operator diagnostics survive a
// coordinator restart.
@@ -217,7 +234,14 @@ type Task struct {
// current one, oldest first. A superseded plan stays queryable: the
// verification recorded against it is provenance, not garbage.
PlanHistory []string `json:"plan_history,omitempty"`
LastError string `json:"last_error,omitempty"`
// PlanMismatch is the contradiction that reopened this task's phase. The
// planning session convened by a replan has to be told what it is there to
// fix, and the implementer that found it is gone by then (F66). It is
// cleared when a replacement plan is accepted, or when the stop it caused
// is answered, because after either it is history rather than a live
// instruction.
PlanMismatch *PlanMismatch `json:"plan_mismatch,omitempty"`
LastError string `json:"last_error,omitempty"`
}
// ReviewRef binds a sealed review artifact to one commit.
+7
View File
@@ -55,6 +55,13 @@ type PlanPhaseRecord struct {
// EvidenceRef is the CAS ref of the captured command output.
EvidenceRef string `json:"evidence_ref,omitempty"`
At time.Time `json:"at"`
// ManualAtSHA is the tree a human was actually looking at when they signed
// this phase off. A manual check on most projects is a human reading
// output, so a sign-off establishes something about one tree and nothing
// about the next one (F63). Rerunning the automated half re-establishes it
// at the new commit; the manual half has to be given again, and this is
// what makes the difference visible instead of assumed.
ManualAtSHA string `json:"manual_at_sha,omitempty"`
}
// Stale reports whether the tree has moved since this phase was verified. A
+8
View File
@@ -174,6 +174,14 @@ type Session struct {
// its §6.1 handoff (HandoffFile) — avoids re-sending the same prompt
// every tick while Release keeps waiting for the file to appear.
HandoffRequested bool `json:"handoff_requested,omitempty"`
// HandoffRequestedAt stamps that prompt. A request nobody answers used to
// end as an ordinary idle expiry, indistinguishable from an agent that
// never started (F62); the stamp is what makes the wait bounded and the
// giving-up causal.
HandoffRequestedAt time.Time `json:"handoff_requested_at,omitempty"`
// HandoffRetried records that the request was re-sent once, so a session
// waiting on an answer is not re-prompted every tick.
HandoffRetried bool `json:"handoff_retried,omitempty"`
// HandoffReason is selected by the coordinator when it asks for the
// semantic report. The checkout worker, rather than the harness, copies
// it into the canonical handoff it seals at release time.
+19 -12
View File
@@ -58,18 +58,27 @@ func RequestHumanDecision(s *store.Store, project registry.Project, taskID strin
return blockTask(s, t, domain.BlockReasonHumanDecision, req.Render(), &req)
}
// fenceToLease binds a lifecycle event to the lease that is producing it.
// Store.Append fences every lifecycle event on a leased task against the
// current owner and epoch, so an event that omits them is a conflict rather
// than a block. Every coordinator-side stop goes through here: two of them did
// not, and both failed silently against a live lease (F65). A task stops for a
// human only while some session is running, so the leased case is the only one
// that ever mattered.
func fenceToLease(payload map[string]any, t domain.Task) {
if t.Lease == nil {
return
}
payload["harness_id"] = t.Lease.HarnessID
payload["lease_epoch"] = t.Lease.Epoch
}
func blockTask(s *store.Store, t domain.Task, reason domain.BlockReason, blocker string, req *domain.DecisionRequest) (domain.Event, error) {
payload := map[string]any{
"blocker": blocker, "block_reason": string(reason),
"lifecycle_phase": "awaiting_human",
}
if t.Lease != nil {
// Store.Append fences every lifecycle event on a leased task against
// the current owner and epoch. A question from a session that no
// longer owns the task is a conflict, not a block.
payload["harness_id"] = t.Lease.HarnessID
payload["lease_epoch"] = t.Lease.Epoch
}
fenceToLease(payload, t)
if req != nil {
payload["decision_request"] = req
}
@@ -107,11 +116,9 @@ func ResumeAnsweredBlockers(s *store.Store) ([]domain.Event, error) {
if t.State != domain.StateBlocked {
continue
}
switch t.BlockReason {
case domain.BlockReasonHumanDecision, domain.BlockReasonTrajectoryGate:
default:
// operator_required is deliberately not resumed by a reply. An
// operator decides when a task that spent its budget continues.
// operator_required is deliberately not resumed by a reply. An
// operator decides when a task that spent its budget continues.
if !domain.BlockReasonAwaitsReply(t.BlockReason) {
continue
}
if !blockerAnswered(s, t.ID, t.BlockReason) {
+4 -2
View File
@@ -154,11 +154,13 @@ func blockForPlanMismatch(s *store.Store, taskID string, m domain.PlanMismatch)
packet += "- evidence: " + oneLine(e) + "\n"
}
packet += "\nReply to say how to proceed. Your reply becomes a recorded decision and outranks the plan. If it resolves the contradiction, the task resumes on the same plan; say so explicitly if you want the plan rewritten instead.\n"
b, err := json.Marshal(map[string]any{
payload := map[string]any{
"blocker": packet,
"block_reason": string(domain.BlockReasonPlanMismatch),
"lifecycle_phase": "awaiting_human",
})
}
fenceToLease(payload, t)
b, err := json.Marshal(payload)
if err != nil {
return err
}
+28
View File
@@ -108,6 +108,11 @@ func TestReplanKeepsTheOldPlanUntilAReplacementIsSealed(t *testing.T) {
if len(during.PlanHistory) != 0 {
t.Fatalf("the plan was moved to history early: %v", during.PlanHistory)
}
// F66: the planning session this reopen convenes has to be told what it is
// there to fix, and the implementer that found it is gone by then.
if during.PlanMismatch == nil || during.PlanMismatch.PhaseID != "phase-1" {
t.Fatalf("the contradiction did not survive to the phase it reopened: %+v", during.PlanMismatch)
}
}
// Sealing the replacement is the moment the old plan is superseded. Progress
@@ -151,6 +156,11 @@ func TestSealingTheReplacementSupersedesThePlanAndItsProgress(t *testing.T) {
if _, err := s.Artifact(oldRef); err != nil {
t.Fatalf("the superseded plan is unreadable: %v", err)
}
// The replacement answers the contradiction, so it stops being a live
// instruction and stays in the log as history.
if after, _ := s.Task(id); after.PlanMismatch != nil {
t.Fatalf("a settled contradiction is still live: %+v", after.PlanMismatch)
}
assertPhase(t, s, id, domain.WorkPhaseImplement)
}
@@ -215,7 +225,25 @@ func TestHumanAnswerResumesTheSamePlanWithoutResealing(t *testing.T) {
if !PlanMismatchAnswered(s, id) {
t.Fatal("the human answered and the task is still waiting")
}
// F64: asserting the predicate is not asserting the resume. This test
// passed for as long as the predicate had no caller, while a task blocked
// on a mismatch stayed blocked forever however the human replied.
if events, err := ResumeAnsweredBlockers(s); err != nil || len(events) != 1 {
t.Fatalf("an answered mismatch did not return to the queue: events=%v err=%v", events, err)
}
after, _ := s.Task(id)
if after.State != domain.StateQueued {
t.Fatalf("state = %s after the human answered, want queued", after.State)
}
// F67: the stop is over, so the projection must not keep reporting it.
if after.BlockReason != "" || after.Blocker != "" {
t.Fatalf("a resumed task still reports its blocker: %q %q", after.BlockReason, after.Blocker)
}
// The answer outranks the plan and stands as an ordinary decision, so the
// contradiction it settled is not carried into later sessions either.
if after.PlanMismatch != nil {
t.Fatalf("an answered contradiction is still live: %+v", after.PlanMismatch)
}
if after.PlanRef != planRef {
t.Fatal("answering the question replaced the plan")
}
+21 -5
View File
@@ -124,7 +124,12 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
// was established.
record.Status = domain.PlanPhaseAwaitingManual
}
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID) {
// Carry the confirmed tree forward as provenance. Without it a second
// rerun would compare against nothing and re-inherit the sign-off.
if prior, ok := t.PlanPhase(phaseID); ok {
record.ManualAtSHA = prior.ManualAtSHA
}
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID, record) {
record.Status = domain.PlanPhaseVerified
}
if ref, err := s.PutArtifact(verificationEvidence(runs)); err == nil {
@@ -135,7 +140,7 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
payload := map[string]any{
"plan_ref": record.PlanRef, "phase_id": record.PhaseID, "status": string(record.Status),
"commands": record.Commands, "exit_codes": record.ExitCodes, "at_sha": record.AtSHA,
"evidence_ref": record.EvidenceRef, "at": record.At,
"evidence_ref": record.EvidenceRef, "at": record.At, "manual_at_sha": record.ManualAtSHA,
}
if t.Lease != nil {
payload["harness_id"], payload["lease_epoch"] = t.Lease.HarnessID, t.Lease.Epoch
@@ -149,9 +154,20 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
}
// manuallySignedOff reports whether a human has already approved this exact
// phase of this exact plan. The subject carries both, so a later "looks good"
// on an unrelated thread cannot satisfy a gate nobody was discussing.
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string) bool {
// phase of this exact plan, against the tree this run examined. The subject
// carries plan and phase, so a later "looks good" on an unrelated thread
// cannot satisfy a gate nobody was discussing.
//
// The tree matters as much as the subject (F63). A sign-off is a human saying
// they read what this code prints; an edit afterwards can change exactly that.
// A record whose ManualAtSHA names a different commit is therefore not signed
// off, and waits for the human again. A sign-off given before any run has no
// confirmed tree to compare against and still counts, which keeps the ordinary
// ordering unchanged.
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string, record domain.PlanPhaseRecord) bool {
if record.ManualAtSHA != "" && record.ManualAtSHA != record.AtSHA {
return false
}
intent, err := s.EffectiveIntent(t.ID)
if err != nil {
return false
+66 -1
View File
@@ -13,6 +13,7 @@ import (
const shaOne = "1111111111111111111111111111111111111111"
const shaTwo = "2222222222222222222222222222222222222222"
const shaThree = "3333333333333333333333333333333333333333"
func planProject() registry.Project {
p := registry.Project{
@@ -27,6 +28,11 @@ func planProject() registry.Project {
func planWith(t *testing.T, markdown string) (*store.Store, registry.Project, string) {
t.Helper()
s, id := phaseStore(t)
// Leased, because everything these tests drive comes from a live implement
// session. Skipping it hid F65: two coordinator-side stops omitted the
// fencing fields Store.Append requires on a leased task, and every test
// passed because no test ever leased one.
lease(t, s, id)
project := planProject()
if _, err := AdvanceWorkPhase(s, project, id, nil); err != nil {
t.Fatal(err)
@@ -251,6 +257,14 @@ func TestLegacyPlanIsExplicitlyNonProgressable(t *testing.T) {
// signOff records a human decision bound to one phase of one plan, which is
// the only thing that satisfies a manual verification gate.
func signOff(t *testing.T, s *store.Store, taskID, subject string) {
t.Helper()
signOffFrom(t, s, taskID, subject, "signoff-"+subject)
}
// signOffFrom names the comment the sign-off came from. Two sign-offs on one
// subject are a real sequence once a rerun sends a phase back to the human,
// and provenance is unique per comment.
func signOffFrom(t *testing.T, s *store.Store, taskID, subject, externalID string) {
t.Helper()
task, _ := s.Task(taskID)
if err := s.Append(domain.Event{
@@ -259,9 +273,60 @@ func signOff(t *testing.T, s *store.Store, taskID, subject string) {
Payload: mustJSONBytes(t, map[string]any{
"decision_id": domain.NewID(), "kind": "answer", "subject": subject,
"value": "manual steps confirmed",
"source": map[string]any{"provider": "gitea", "external_id": "signoff-" + subject},
"source": map[string]any{"provider": "gitea", "external_id": externalID},
}),
}); err != nil {
t.Fatalf("sign off: %v", err)
}
}
// F63, found live on run 19. A manual sign-off says a human read what this
// code prints. An edit afterwards can change exactly that, so rerunning the
// automated half at a new commit must not carry the human half with it.
func TestASignOffDoesNotSurviveTheTreeItWasGivenAgainst(t *testing.T) {
s, project, id := planWith(t, twoPhasePlan)
run := []VerificationRun{{Command: []string{"go", "test", "./internal/..."}, ExitCode: 0}}
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaOne, run); err != nil {
t.Fatal(err)
}
task, _ := s.Task(id)
signOff(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"))
task, _ = s.Task(id)
rec, _ := task.PlanPhase("phase-2")
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaOne {
t.Fatalf("sign-off did not bind to the tree it read: %+v", rec)
}
// The tree moves and the phase is re-verified. The commands pass again;
// the human has not seen the new output.
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaTwo, run); err != nil {
t.Fatal(err)
}
task, _ = s.Task(id)
rec, _ = task.PlanPhase("phase-2")
if rec.Status != domain.PlanPhaseAwaitingManual {
t.Fatalf("status = %q at a tree the human never saw, want awaiting_manual_verification", rec.Status)
}
if rec.ManualAtSHA != shaOne {
t.Fatalf("the confirmed tree was lost: %+v", rec)
}
// A second rerun must not re-inherit it either, which is what carrying
// ManualAtSHA forward is for.
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaThree, run); err != nil {
t.Fatal(err)
}
task, _ = s.Task(id)
rec, _ = task.PlanPhase("phase-2")
if rec.Status != domain.PlanPhaseAwaitingManual {
t.Fatalf("a second rerun re-inherited the sign-off: %q", rec.Status)
}
// Signing off again, on the tree that is now current, verifies it.
signOffFrom(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"), "signoff-second")
task, _ = s.Task(id)
rec, _ = task.PlanPhase("phase-2")
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaThree {
t.Fatalf("a fresh sign-off did not verify the current tree: %+v", rec)
}
}
+4 -2
View File
@@ -67,11 +67,13 @@ func raiseTrajectoryGate(s *store.Store, t domain.Task, from, to domain.WorkPhas
if err != nil {
return err
}
b, err := json.Marshal(map[string]any{
payload := map[string]any{
"blocker": packet,
"block_reason": string(domain.BlockReasonTrajectoryGate),
"lifecycle_phase": "awaiting_human",
})
}
fenceToLease(payload, t)
b, err := json.Marshal(payload)
if err != nil {
return err
}
+28
View File
@@ -272,6 +272,9 @@ func (s *Store) apply(e domain.Event) error {
}
if p.Subject == domain.PlanPhaseSubject(rec.PlanRef, rec.PhaseID) {
t.PlanProgress.Phases[i].Status = domain.PlanPhaseVerified
// Record which tree the sign-off was about, so a later run
// at a different commit cannot inherit it (F63).
t.PlanProgress.Phases[i].ManualAtSHA = rec.AtSHA
t.Version = e.Version
s.replaceTask(e.TaskID, t)
}
@@ -309,6 +312,9 @@ func (s *Store) apply(e domain.Event) error {
}
t.PlanRef = p.ArtifactRef
t.PlanProgress = nil
// The replacement answers the contradiction. Carrying it
// further would put a solved problem in every later context.
t.PlanMismatch = nil
}
}
t.WorkPhase = p.Phase
@@ -484,6 +490,19 @@ func (s *Store) apply(e domain.Event) error {
t.Version = e.Version
s.replaceTask(e.TaskID, t)
return nil
case domain.EventPlanMismatchRecorded:
// Projected so the phase this reopens can be told what reopened it.
// The event is the record; this is the live instruction derived from
// it, and it stops being live as soon as a replacement seals or the
// stop is answered.
var m domain.PlanMismatch
if err := json.Unmarshal(e.Payload, &m); err != nil {
return err
}
t.PlanMismatch = &m
t.Version = e.Version
s.replaceTask(e.TaskID, t)
return nil
case domain.EventPlanPhaseVerified:
var pp domain.PlanPhaseRecord
if err := json.Unmarshal(e.Payload, &pp); err != nil {
@@ -585,8 +604,17 @@ func (s *Store) apply(e domain.Event) error {
// A question only stands while the task is blocked on it. Afterwards the
// answer is an ordinary standing decision and the log still holds the
// question, so keeping it on the task would put it in every later context.
//
// The blocker itself has the same lifetime and was outliving it (F67). A
// task resumed from a plan mismatch kept reporting block_reason
// plan_mismatch while it ran, so every surface reading the projection
// showed a stop that had already been answered.
if t.State != domain.StateBlocked {
t.DecisionRequest = nil
t.Blocker, t.BlockReason = "", ""
// The human's answer outranks the plan and stands as an ordinary
// decision, so the contradiction it settled is no longer live.
t.PlanMismatch = nil
}
if phase, ok := p["lifecycle_phase"].(string); ok && phase != "" {
t.LifecyclePhase = phase
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

+635
View File
@@ -0,0 +1,635 @@
# orchestra web control ui
status: mockup specification
app: orchestra
shared design system: ethos
accent: signal violet `#8F7AE5`
motif: routing fork
primary operator: single infrastructure operator
## 1. purpose
orchestra's web ui is an operator console for understanding and steering agent work without reconstructing state from logs, tmux sessions, gitea, or the event stream.
it has three jobs:
1. show what needs the operator now.
2. show what orchestra believes and what an agent is actually doing.
3. expose intervention and forensic detail without making the default screens noisy.
it is not a generic project-management product, an analytics dashboard, or a chat-first interface.
## 2. ethos application
orchestra inherits the ethos shell and neutral system unchanged.
### fingerprint
- accent: signal violet `#8F7AE5`
- motif: routing fork
- accent use: active navigation, focus, selected phase, primary operator action, routing/transition marks
- motif use: app mark, phase transitions, routing state, empty states, one hero/detail moment
suggested token block:
```css
[data-app="orchestra"] {
--accent: #8F7AE5;
--accent-hi: #A291EC;
--accent-dim: rgba(143, 122, 229, 0.14);
--accent-line: rgba(143, 122, 229, 0.32);
--accent-glow: rgba(143, 122, 229, 0.22);
}
```
### type
- geist sans: labels, task names, explanations, buttons, headings, operator-authored prose
- geist mono: task ids, epochs, worker ids, pane ids, timestamps, percentages, context counts, revisions, paths, refs, sha values, lease durations
- machine values must never silently use sans for visual convenience
### surfaces
- page room: `--bg-0`
- normal panel: `--bg-1`
- selected/raised content: `--bg-2`
- hover: `--bg-3`
- borders: 1px `--line`, `--line-hi` only where selection or risk needs stronger separation
- no blur or glass
- accent never washes a whole content panel
### motion
all ui transitions use ethos mechanical motion: 130170 ms, `cubic-bezier(0.2, 0, 0, 1)`, no spring or bounce.
## 3. shared shell
### desktop
- 64px vertical icon rail
- 56px top bar
- main content fills remaining viewport
- global `cmd+k` command/search surface in the top bar
- right side of top bar shows only high-value machine state: orchestra revision, live state, current time
primary navigation:
1. dashboard
2. tasks
3. decisions
4. workers
5. projects
6. review
7. settings
terminal is entered from a task or worker and does not need permanent primary navigation.
### mobile
- rail becomes bottom navigation with no more than five primary items
- secondary pages move behind task/project drill-down or header menus
- machine readouts relocate into detail screens rather than disappearing
- terminal remains a dedicated full-screen view
## 4. information hierarchy
orchestra should visually distinguish five classes of information.
| class | meaning | treatment |
|---|---|---|
| operator authority | human decision, correction, constraint | high prominence, accent signal |
| verified orchestra state | lifecycle, lease, phase, review, refs | neutral surfaces, machine values in mono |
| agent claims | proposed plan, handoff text, self-reported risk | visibly labeled as agent-supplied |
| historical/superseded | stale plan, previous handoff, old decision | muted, never visually dominant |
| fault/attention | blocked, refused, stale, mismatch | status color + explicit reason |
raw events and implementation details are drill-down evidence, not default-page content.
## 5. dashboard
mockup: `final/01-dashboard.png`
purpose: answer "what needs me, what is running, and is the system healthy?"
### layout
three primary regions only:
1. **needs your attention**
- trajectory gate
- blocking human decision
- pr awaiting review
- closed-without-merge/operator-required state
- one line explaining why each item needs the operator
2. **running tasks**
- task name/id
- current phase
- worker/harness
- last real progress
- context occupancy
- lease timing
- compact routing-fork phase indicator
3. **system state and capacity**
- workers online/available
- quota status
- active leases
- router queue/rejection state
- source reconciliation
- herdr health
### rules
- no disk, cpu, ram, or generic host-monitor metrics unless they directly block orchestration
- no vanity charts
- health should show actual counts/times, not vague "running" indicators
- dashboard actions should navigate to the relevant task/review/worker rather than becoming an operator-control surface itself
## 6. tasks list
mockup: `final/04-tasks.png`
purpose: browse all work under orchestra and locate a task quickly.
### groups
- needs attention
- running
- waiting/queued
- completed
- failed
### row fields
- task title/id
- project
- current phase
- compact routing-fork phase path
- worker/harness when assigned
- last update/progress
- context occupancy for active sessions
- status
### filters
- project
- phase
- worker/harness
- status
- attention state
avoid turning this into a kanban board. orchestra's lifecycle is stateful and event-backed; the list should expose state, not encourage arbitrary drag-and-drop mutation.
## 7. task detail
mockup: `final/02-task-detail.png`
purpose: provide one calm page answering "what are we doing?", "what is happening?", and "where are we in the workflow?"
### top area
- title
- task id
- project
- current phase
- steer/correct primary action
- secondary actions menu
### what we're doing
show only:
- goal
- acceptance summary
- latest effective human decisions when present
- active constraints when present
full contract, all decisions, and historical versions open in drill-down views.
### live execution
show only:
- worker/harness
- pane
- lease remaining
- context occupancy
- last progress
- one-line current activity
links:
- open live pane
- worker logs
- launch context
- git diff
### workflow
render the ace-fca path visibly:
`frame → research → plan → implement → review → pr → merged`
completed phases are quiet green, current phase uses the orchestra accent, future phases remain neutral.
phase detail, artifacts, refusal history, and raw events belong in drill-downs rather than the default task screen.
## 8. live pane / terminal
mockup: `final/03-terminal-live-pane.png`
purpose: show the actual tmux/herdr pane, not a reconstructed log view.
### behavior
- read-only by default
- exact pane dimensions and terminal rendering preserved when possible
- fit mode may scale visually but must not alter the underlying pane
- scrollback inspection must not disturb the live cursor
- `take control` explicitly enables keyboard forwarding
- strong visual state when control is captured
- escape or a fixed shortcut releases control
### why this screen exists
it must reveal real tui state such as:
- input still sitting in the editor
- queued input
- paste placeholders
- permission/approval dialogs
- `/clear`
- `@HANDOFF.md`
- agent busy/idle presentation
- tmux status line and pane identity
### surrounding chrome
keep it minimal:
- worker/pane identity
- read-only/control state
- fit/copy/scrollback controls
- lease epoch/timing
- context occupancy
- last verified progress
- pane/worker health
- small live event strip for orchestra-originated interactions and confirmations
when a pane is lost, preserve the last captured frame and state the exact loss time and lease consequence.
## 9. decisions list
mockup: `final/05-decisions.png`
purpose: inspect durable human authority and approvals across tasks.
this is not the human attention queue; that is review.
### list fields
- decision id
- short decision text
- task
- phase
- source/provenance
- requested/recorded time
- status: active, superseded, waiting, approved, rejected, auto
- operator/actor
### filters
- active/superseded
- kind: decision, correction, constraint, answer, scope change
- project
- task
- actor/source
### detail
opening a decision should show:
- exact value
- subject
- provenance
- supersedes/superseded-by links
- event sequence
- effective authority impact
machine provenance is mono; the human decision text remains sans.
## 10. workers
mockup: `final/06-workers.png`
purpose: answer "where can work run and is that execution machinery usable?"
### worker list fields
- worker/harness identity
- online/idle/busy/offline/degraded
- harness type
- current task when leased
- phase
- lease or heartbeat age
intentionally omit generic infrastructure metrics such as disk, memory, cpu, and uptime unless orchestra directly uses them for eligibility.
### worker detail
- current session/task
- pane id
- lease epoch/remaining
- last verified progress
- worker revision
- herdr status
- last heartbeat
- projects declared by the worker
- declared capabilities if orchestra actually consumes them
### capabilities
capabilities must come from worker registration/project configuration, never be guessed from host inspection.
the ui labels them **declared capabilities** and can expose registration provenance.
### registration
registration is worker-driven:
1. worker starts with coordinator credentials/configuration
2. worker announces identity, revision, harnesses, projects, capacity and declared capabilities
3. coordinator records it and heartbeat updates liveness
`add worker` should therefore be a registration guide/bootstrap flow, not a magical browser-side creation of a remote process.
## 11. projects
mockup: `final/07-projects.png`
purpose: group long-running repository/workflow configuration and its active work.
### project list
- project name
- short description
- state
- active/total task count
- attention/decision count
- last meaningful activity
avoid aggregate artifact counts or infrastructure metrics unless they help choose an action.
### project detail
- description
- source/repository identity
- configured worker/harness affinity
- workflow phases
- trajectory-gate policy
- quality gate
- human reconcile source
- safe-operation policy
- active tasks
- recent meaningful events
configuration edits belong in the project detail, while global defaults remain in settings.
## 12. review
mockup: `final/08-review.png`
purpose: the operator inbox: **what needs me to look at something and judge?**
workers show machinery. review shows human attention.
### review item types
- trajectory decision
- human blocker / decision request
- code/pr review
- ai review requiring operator override
- operator-required lifecycle state
- closed pr without merge
- information/fyi, optionally hidden by default
### list fields
- type
- task/item
- why it is here
- project
- priority
- age
worker/pane is secondary metadata, not a primary column.
### detail drawer/page
show the minimum evidence needed to act:
- why this needs the operator
- effective task goal/constraints relevant to the decision
- proposed options when applicable
- relevant plan/diff/review evidence
- provenance and age
- comments/context drill-down
primary action names must match the lifecycle action: make decision, answer blocker, review pr, request changes, retry, etc.
snooze/defer may exist for attention management but must not silently mutate underlying task truth.
## 13. settings
mockup: `final/09-settings.png`
purpose: configure orchestra itself, project-independent defaults, integrations, policies, and operator/security settings.
### tabs
1. general
2. appearance
3. notifications
4. access & security
5. integrations
6. agents & tools
7. policies
8. advanced
### general
- instance name/environment
- locale/time display
- default task preferences
- default workflow policy
- event/artifact retention
- storage/backup state only if orchestra owns it
### appearance
- theme: system/dark/light
- ethos remains fixed; no arbitrary shell redesign controls
- accessibility and reduced motion
### access & security
- surface tokens/status
- trusted review actors
- ignored actors
- operator sessions
- agent-surface auth
- secret/config source status without exposing secret values
### integrations
- gitea
- vikunja
- notification sinks
- source reconciliation state
show endpoint/identity and last successful interaction in mono.
### agents & tools
- harness definitions
- adapter/runtime configuration
- tmux/herdr ownership
- default launch behavior
- context occupancy thresholds
### policies
- work phase defaults
- trajectory gate
- bounded decision request budget
- retry policy
- reconcile failure handoff threshold
- quality/review/submission policy
### advanced
- event-store diagnostics
- snapshots/replay
- raw configuration
- migration/status information
- destructive operator actions in a visually separated danger zone
## 14. global command surface
`cmd+k` is the shared ethos command surface and should handle navigation plus safe operator actions.
examples:
- open task by id/title
- open worker/pane
- show decisions for task
- steer/correct active task
- answer current blocker
- retry failed task
- open review item
high-risk lifecycle actions require explicit confirmation and should not be the first fuzzy-search result for ordinary text.
## 15. status grammar
status is compact and consistent across screens.
recommended semantic colors:
- green: verified healthy/completed/accepted
- violet: current/selected/orchestra-controlled active state
- amber: waiting/degraded/operator attention
- red: failed/refused/danger
- gray: idle/unknown/historical
color never carries the entire meaning; every state also has text/iconography.
## 16. evidence drill-down
any important derived claim should be traceable without ssh.
from relevant detail screens the operator should be able to reach:
- event(s) that established the state
- exact generated launch context
- sealed research/plan/review/submission artifact refs
- git sha/ref
- worker journal observations
- input confirmation receipt
- pane capture/terminal
- source comment/review provenance
raw evidence is accessible but not visible by default.
## 17. copy rules
- labels describe what the operator controls, not implementation class names
- machine values remain exact and honest
- errors state what failed and what consequence follows
- no "something went wrong"
- no spinner where a queue depth, heartbeat age, poll timestamp, retry time, or progress value exists
examples:
- `source reconcile failed · retry at 14:06:00`
- `worker heartbeat 42s old · ineligible after 60s`
- `phase request refused · next phase is research`
- `pane lost at 13:42:18 · lease expires in 09:14`
## 18. responsive behavior
### tablet/mobile priorities
keep visible:
1. attention state
2. task title/phase
3. latest human authority
4. live execution state
5. primary action
collapse or drill down:
- long acceptance lists
- complete workflow history
- detailed lease metadata
- worker journal
- raw events
- project configuration
terminal becomes its own full-screen route and should not be squeezed into a card.
## 19. implementation order
recommended web-ui build order:
1. ethos shell + orchestra tokens/motif
2. dashboard
3. tasks list
4. task detail
5. live pane
6. review inbox
7. decisions list/detail
8. workers
9. projects
10. settings
11. global command surface
12. evidence/raw-event drill-down
build each screen against real orchestra api data rather than static ui-only state as early as practical.
## 20. verification
for every screen:
- render and visually inspect pixels
- check desktop and <=640px mobile layout
- assert no horizontal overflow
- verify mono/sans split visibly
- verify accent is signal-only
- verify no blur/backdrop-filter exists
- verify empty/error/loading states show real machine state
- compare implemented screen against the corresponding mockup as direction, not pixel-perfect contract
mockups communicate hierarchy, density, and interaction intent. orchestra state and the ethos design laws are the actual specification.