25 Commits

Author SHA1 Message Date
kami f25964c18e Record slice B as proven with two unexercised branches
Neither will be manufactured: a second task hitting the same failure
exercises the eligibility transition, and a genuine repair exercises manual
intervention provenance.

Also records why the reasons check earned its place. An inference drawn
from aggregate numbers was wrong, and the pure explanation contradicted it
with counted facts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-30 14:57:43 +04:00
kami 0aefe021b0 Explain a debt refusal with facts about the item
Reasons are the auditable explanation of a pure decision, so one that
describes the architecture rots the moment the architecture moves. The
operational refusal asserted that manual interventions were recorded by no
event type: true when written, false the day OperatorInterventionRecorded
landed, and still printed under every refusal after that.

It now reports the counts and the thresholds they missed. Why a count is
zero is not this function's business, since no intervention happening, none
being recorded, and none being migrated all read the same from here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-30 14:57:25 +04:00
kami 982741fe97 Say which operator actions are debt and which are just operations
An intervention is an action required to recover, repair, unblock or correct
behaviour that should have proceeded on its own. A deploy restart, a planned
upgrade and burn-in setup are none of those, and recording them would inflate
operator cost with work Orchestra is expected to need.

Policy rather than schema. worker_restart is the one kind that spans both
sides, and it stays one kind until someone actually misuses it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-30 14:29:17 +04:00
kami 42c5f07844 Write down the rule both slice B defects broke
A projection must never manufacture provenance to make evidence easier to
classify. task=None is better than a confident lie.

Also records that signatures are frozen at write time, so an old event never
changes meaning because normalization code changed. Repairing historical
signatures is an explicit migration, not a different projection result from
the same log.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-30 14:23:41 +04:00
kami 79d20534b5 Stop the ring's history from manufacturing recurrence
Both defects came from the first real run of slice B against live data, and
neither was visible in a unit test written from the design.

The ring is a bounded history, not a set of live conditions. A quiet timeout
closed an incident, its entry stayed in the ring because nothing evicts it,
and every later heartbeat opened the same incident again: three signatures,
four incidents each, from failures that never happened twice. An incident
now opens only when the entry actually advances past what was already
accounted, and the high-water mark survives the close.

The ring also outlives the work it describes, so attributing its entries to
whatever the worker is running now invented an association. The task is read
out of the message, and only a failure that names no task belongs to the
current lease. An incident that names an older task has no live lease to
bound it, so it closes on quiet timeout rather than on the next epoch change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-30 14:20:47 +04:00
kami 74cad5d374 Collapse a pane name's task id in an observation signature
The ledger's first live run showed it: a pane name carries the task id in
lower case, so "phase rotation ...: pane orchestra-<task>-<sha>:1.0 still
holds input" signed differently on every task and could never accumulate
recurrence across them. The id pattern is case-insensitive now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-30 06:55:51 +04:00
kami e0601296e0 Record what an operator repaired, and let debt count incidents
Slice B, second half. OperatorInterventionRecorded is the one command for
saying "I fixed this by hand": a manual repair happens outside Orchestra by
definition, so the only honest way to have the evidence is for the person
who made it to state it. Inferring "an operator probably intervened" from a
gap would put guesses into the record the ledger is built from.

The debt projection now consumes both new kinds. A closed incident is one
observation carrying its repeat count as intensity, so recurrence stays a
count of independent incidents: 301 repeats on one lease and 2 on another
is a recurrence of two with an intensity of 303, not a recurrence of 303.

Both kinds were previously reported as holes in the system. They are
ordinary evidence now, so their absence from a history is a fact about that
history, and the gap list says so.

The worker also stamps a per-process incarnation on registration and every
heartbeat. Nothing else on the wire distinguishes a restarted worker from a
running one, and an incident cannot outlive the process that reported it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-30 06:53:26 +04:00
kami 438c1d6df3 Make worker observations durable as incidents, not as symptoms
Slice B, first half. The F18 ring is bounded, lossy and local, so the debt
ledger reported it as a gap about itself. Two events make it durable:
ObservationIncidentOpened at first sight, appended immediately so a
coordinator that dies mid-incident still leaves the fact that it existed,
and ObservationIncidentClosed carrying the aggregate.

The rules are what matter. Repeats update the aggregate and append nothing,
because run 11's 409 loop was one incident with an intensity of 301 rather
than 301 pieces of evidence. Absence from the ring closes nothing, since a
bounded history evicts as easily as it recovers. An incident is scoped to
its lease and closes on lease end, epoch change, worker restart, or, for
observations with no lease to bound them, on last_seen going stale. A ring
entry that is evicted and recreated accumulates: 34 then 3 is 37.

Also collapses three drifted copies of the list of events that carry no
task into one predicate. Adding a type to two of them left it rejected by
the third, which is how the first version of this failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-30 06:35:17 +04:00
kami 76da8c40b7 End the implementer's trajectory when Orchestra reopens the phase
Run 20 showed the leak precisely: the reopen rotated the session, and 31
seconds later the planning session that replaced it recorded phase-2 of the
plan being replaced. The implementer's .orchestra/plan-progress.json had
outlived it in the worktree, and the successor executed it.

Two changes. Phase verification now belongs to the implement phase and is
refused anywhere else, which covers every worker and every path rather than
the one that produced it. The rotation also drops the ended session's
request files, so a successor never inherits work from a trajectory
Orchestra has already invalidated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-30 05:46:11 +04:00
kami 575e3ef87e Record run 21: F66 proven, and why round one did not count
The first fix wrote the projection and lost it at the rotation the reopen
causes, so the planner still got nothing. Both rounds passed their unit
tests; what separated them was where the assertion sat in the sequence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 22:03:51 +04:00
kami 75396963ef Keep the contradiction alive across the rotation the reopen causes
The clearing rule sat below the reducer switch, so it ran for every event
rather than for the correction it was written for. TaskReleased found the
task unblocked and erased the contradiction, which is the rotation the
reopen itself causes: the planning session launched one lease later and was
told nothing again.

Scoped to TaskCorrected, and the store test walks the real sequence
(mismatch, reopen, release, lease) rather than reading the projection at
the moment it is written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 21:05:01 +04:00
kami edf00761fd Tell the planner what it was convened to fix
F66, found live on run 20. A replan reopens the plan phase and the session
it convenes was given the ordinary plan brief and an empty template: no
mention that a plan already exists, what contradicted it, which phase, what
evidence, or that its output supersedes an accepted plan. All of it was
already durable on PlanMismatchRecorded and none of it reached the agent,
so nothing stopped the replacement from sealing with the same
contradiction in it.

The contradiction is now projected onto the task and rendered above the
sealed artifacts, because it changes how they should be read. Its lifetime
is bounded at both ends: a sealed replacement answers it, and so does a
human reply to the stop it caused. Only the phases a mismatch can reopen
render it, since an implementer looks at the code rather than at an account
of the last contradiction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:47:44 +04:00
kami 1330ad9943 Stop reporting a blocker the task is no longer blocked on
F67. TaskCorrected cleared the standing question when a task resumed and
left Blocker and BlockReason in place, so task 29 ran through implement,
review and submission still reporting block_reason plan_mismatch. Every
surface that reads the projection rather than the event log showed a stop
that had already been answered.

The blocker has the same lifetime as the question beside it, and is now
cleared with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:41:36 +04:00
kami cb80dd1952 Record run 20: the mismatch and replan ladder holds, and F66
Both halves proven live. A human_decision mismatch stops the task, the
blocker packet states the contradiction, and a Gitea reply resumes the same
plan in 55 seconds with no reseal. A replan reopens into plan while the old
plan stays accepted with its progress, and sealing the replacement moves
plan_ref, retains the old one in history, stops the old progress counting,
and hands the successor the replacement whole.

F66 recorded: the planning session a replan convenes is told nothing about
the contradiction it exists to fix, though all of it is durable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:39:04 +04:00
kami de18f372d3 Fence the two coordinator-side stops nobody had leased
F65, found live on run 20. A plan mismatch asking for a human decision
recorded its observation, then failed to block the task: Store.Append fences
every lifecycle event on a leased task against the current owner and epoch,
and this TaskBlocked carried neither. The task kept implementing while the
contradiction sat durable in the log, and the agent was told its report was
refused. The trajectory gate had the same omission.

The human-decision path already did this correctly and explained why in a
comment. That comment is now a helper all three call.

The tests could not have caught it. planWith never leased its task, so
every plan test ran in a state no agent can be in, which is exactly what
the lease helper's own comment warns against. It leases now, and the
mismatch block test fails without the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:07:23 +04:00
kami bdc0d4d5be Say that a coordinator deploy leaves the console behind
The ethos console sat undeployed for hours while two coordinator deploys
went out, because both used --no-deps orchestra-api and the web image is
built separately. Checking the commit does not catch it; checking the
served bundle does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 20:02:29 +04:00
kami 6cb2f932d8 Go and read the reply to a question Orchestra asked
F64. A plan mismatch that asks for a human decision blocks the task, and
nothing came back. Two independent gaps, either one enough to strand it:
the reconciler ran only before a lease and at a turn boundary, so a blocked
task's reply was never even read, and ResumeAnsweredBlockers listed two
block reasons, not this one. PlanMismatchAnswered had no caller anywhere.

Blocked tasks awaiting a reply are now reconciled on their own loop, the
same reasoning the submitted-work loop above it already uses: a task that
cannot be leased cannot be reconciled behind a pre-lease hook. One
predicate, BlockReasonAwaitsReply, now names the set for both loops so they
cannot drift apart again.

The existing test asserted the predicate and never the resume, which is how
this survived. It asserts the resume now, and fails without the fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 19:57:11 +04:00
kami b5c37f693b Bind a manual sign-off to the tree it was given against
F63, found live on run 19. A manual check on these projects is a human
reading what the code prints. RecordPlanPhaseVerification asked only
whether a sign-off for that plan and phase existed, and one exists forever,
so rerunning a phase's automated checks at a new commit carried the human
half along with it. The rig proved it twice: two operator commits and two
re-verification requests, each coming back verified without anyone looking.

The reducer now records which tree the human confirmed, the record carries
it forward as provenance, and a run whose commit does not match it waits
for the human again. A sign-off given before any run has no confirmed tree
and still counts, so the ordinary ordering is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 19:41:50 +04:00
kami bbf3335857 Record run 19: the manual verification seam, and F63
The keyed seam holds. Automated checks alone leave a phase awaiting, a
generic approving comment satisfies nothing, and the keyed form verifies
exactly the phase it names under the plan ref Orchestra supplies. A
re-verification request reruns the plan's own commands at the new tree and
writes a record bound to it.

F63 is what the rig found. A manual sign-off is consulted by existence
alone, so it survives every later tree the human never saw, while the
automated half is re-established at each new SHA. The proposed fix stamps
the satisfied record's sha on the decision and requires it to match.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 19:25:21 +04:00
kami 99b209ba10 Record run 18: an implement successor inherits verified phase progress
The rung the last two runs missed. A five-phase task with ten named checks
kept the implement phase open long enough to rotate inside it. The successor
picked up in implement and its launch context carried the whole sealed plan,
phase-1 and phase-2 as verified, phase-3 as the first unfinished phase, and
the current human authority.

Unasked-for bonus: the progress block renders SHA staleness itself, naming
the tree each phase was verified against and the tree it is now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 17:48:17 +04:00
kami ab3258833d Record run 17: F62 proven live, from a production trigger
The rig was not needed. An implement to review phase change asked for the
handoff through the ordinary path, and the agent ignored it, which is the
exact shape F62 was written for. Every assertion held: a causal release
rather than an idle expiry, four renewals during the bounded wait, one
resend carrying the original reason, the timeout class at 10m5s, no stale
transaction, and a successor that leased normally a minute later.

Also recorded: a five-phase plan does not lengthen the implement phase, and
the state-file lever cannot be driven with systemctl restart alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 17:38:15 +04:00
kami c587f2cc8d Bound the wait for a handoff nobody answers
F62. The rotation is agent-driven: the worker asks, and the agent must write
its handoff. When the agent never does, renewals stopped on the ordinary
progress gate, the lease expired, and the task lost an attempt with nothing on
record saying a handoff had ever been requested. Run 16 showed only "agent
status idle and pane unchanged", 34 times.

The request is now stamped, and the wait around it is bounded. While Orchestra
is explicitly waiting the lease renews, because a quiet pane is the answer the
agent was told to give. The request is re-sent once after four minutes, with
the reason it was first asked with. At ten minutes the worker nacks with
failure class handoff_unanswered, and the coordinator releases the task naming
that cause instead of letting the lease die as generic idleness.

The class is known to DebtClassForFailureClass, so a harness that ignores
handoff requests accumulates as its own debt item rather than hiding inside
lease_expired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 16:21:35 +04:00
kami 41658aea5b Carry the resume order and the tag's meaning in the handoff
The next session should not rebuild either from the commits. A defect
found in experimental work is not a reason to reopen settled
architecture.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 15:51:42 +04:00
kami 765bf2afc6 Hand off with the release path settled and the UI as a truth detector
F57 to F60 close the release-transaction family, F18 closes the single
last_error slot, the debt ledger exists as a read-only projection, and
the operator console is rebuilt on ethos.

The part worth acting on is what the UI could not do honestly: no
web-facing human-decision write path, no keystroke forwarding, context
occupancy trapped in herdr, project configuration unserved, and no
federated handoff request. F61 and F62 are recorded and unbuilt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 14:59:05 +04:00
kami cbd6b11c49 Record run 16: the rotation rung, half proven
A successor inherits the whole sealed plan. Phase progress is withheld
from a review successor on purpose, so the rung still needs an
implement-phase successor. F62: a requested handoff nobody answers is
invisible, and its only consequence is an expiry indistinguishable from
an idle one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 14:14:37 +04:00
62 changed files with 3455 additions and 70 deletions
+489
View File
@@ -2527,3 +2527,492 @@ transaction and no session behind, on either worker.
Both workers hold no sessions and no release transactions. Deployed pair is
still `3c7cf95` on both halves.
## Run 16, 2026-08-29: the rotation rung, half proven
Task `06G4SWEVP71FYYKAV5FV0ZK5ZG` on `34f3c28`, a three-phase plan
(`78d25730140c`) with automated checks inside policy.
### Proven: a successor inherits the whole sealed plan
The launch context of the session that picked the task up after its rotation
carries the complete plan: overview, current state, desired end state,
non-goals, approach, all three phases with their files, changes and
verification, testing strategy, risks, migration, and ten research citations.
### Not a defect: that successor was not given phase progress
`renderPlanProgress` runs only for `WorkPhaseImplement`
(`internal/agentctx/agentctx.go:321`). This successor picked up in review, and
the code says why: an independent review must reconstruct the change from the
diff rather than inherit the implementer's account. Progress is withheld there
deliberately.
So the rung still needs an **implement-phase** successor. The records exist and
are durable, projected by `Task.PlanPhases()`.
### Why the mid-implement rotation missed
A trivial task spends about four minutes in implement, and all three phases
verified inside two and a half of them, every one against the same tree
`3da86dbf8ad5`. The implementer does the whole change first, then verifies each
phase. The suspend, edit and restart that sets `handoff_requested` takes about
a minute, and it landed after the implementer had already asked for review.
Two mechanics worth keeping:
- **The web handoff action cannot drive this.** `RequestHandoff` needs a local
coordinator and every session here is worker-owned, so it answers 503. That
is the Design A guardrail working, and a fourth UI-exposed gap.
- **A state-file edit does not survive a running worker.** It holds sessions in
memory and writes them back. Suspend, edit, then restart; resuming lets the
old copy win. This cost one attempt to learn.
### F62: a requested handoff that is never answered is invisible
The rotation is agent-driven: the worker asks, and the agent must write
`HANDOFF.md` at a turn boundary. The review agent never did. Renewals stopped,
the lease expired at 10:12:05, and the task requeued having lost an attempt.
Worker health recorded exactly one thing, 34 times:
```text
x34 lease 06G4SWEVP71FYYKAV5FV0ZK5ZG not renewed: agent status idle and pane unchanged
```
Nothing says a handoff was requested and left unanswered. There is no timeout,
no retry, and no observation. The only visible consequence is an expiry that
looks identical to an ordinary idle one. Diagnosing it needed the worker state
file.
The ring earned its keep again: one distinct message with a count of 34, rather
than 34 overwrites of one slot.
**Fixed, not yet proven live.** The request is now stamped
(`herdr.Session.HandoffRequestedAt`) and the wait is bounded by
`watchHandoff` in the worker:
- the lease renews while Orchestra is explicitly waiting, because a quiet pane
is the answer the agent was asked for — the renewal gate's new case, bounded
by `handoffAnswerTimeout`;
- the request is re-sent once at `handoffRetryAfter` (4 minutes), with the same
reason it was first asked with;
- at 10 minutes the worker nacks with failure class `handoff_unanswered`, and
the coordinator emits `TaskReleased reason=handoff_unanswered` rather than
letting the lease die as generic idleness.
`DebtClassForFailureClass` knows the class, so a harness that repeatedly
ignores handoff requests now accumulates in the debt ledger instead of hiding
inside `lease_expired`.
Runtime proof still owed: force a request the agent will not answer, and read
the release event rather than the worker state file.
## Run 17, 2026-08-29: F62 proven live on `c587f2c`
Task `06G4V20T528ZTER7KZBVGNAYXC`, a five-phase plan
(`3b16b77054b9`) on the deployed pair, coordinator and worker both at
`c587f2c`.
### The trigger was production, not a rig
The rig planned for this run was a state-file lever. It was never needed. The
`implement -> review` phase change asked for a handoff through the ordinary
path (`rotateForPhase`, reason `phase_changed`), and the agent ignored it
because this task's brief instructed it to from phase 4 onward. Nothing touched
tmux, no process was stopped, and the pane stayed open for the whole wait.
```text
12:48:09 implement -> review, handoff requested, reason=phase_changed
12:50:09 TaskLeaseRenewed agent idle, pane capture unchanged
12:52:15 pane input confirmed the single resend, 4m06s after the request
12:52:39 TaskLeaseRenewed
12:55:14 TaskLeaseRenewed
12:57:49 TaskLeaseRenewed
12:58:14 TaskReleased reason=handoff_unanswered failure_class=handoff_unanswered
last_error="handoff requested (phase_changed) and unanswered for 10m5s"
12:59:14 TaskLeased successor leases normally
13:02:20 TaskSubmitted
```
Every assertion the operator set for this rung held:
- the release does not look like an ordinary `lease_expired`: it carries
`reason=handoff_unanswered` and `lifecycle_phase=handoff_unanswered`;
- the idle-renewal rule did not kill the wait: four renewals on an idle pane
with no capture movement, each one the gate would have refused before this
commit;
- the resend preserved the reason the request was first made with;
- exactly one resend, not one per tick;
- the timeout produced the class at `10m5s`, one tick past the bound;
- nothing stale was left: `releases: {}`, no session, no lease, and zero tmux
panes matching the id;
- the next attempt leased 60s later and reached `TaskSubmitted`.
The debt projection classified it with no new code beyond the class mapping:
```text
v1:operational:handoff_unanswered:workpc-claude:lease operational important
"handoff requested (phase_changed) and unanswered for 10m5s"
```
**Inference, not observation.** The resent prompt's text was not captured. That
it carried the `phase_changed` wording rests on the code path and the timing,
not on a pane capture. A future rig should publish a capture across the resend.
### The implement-successor rung missed again, and why
A five-phase plan does not lengthen the implement phase. This one ran 12:45:18
to 12:48:09, about 2m50s, with verifications 20 to 40 seconds apart:
```text
12:45:47 phase-1 12:46:07 phase-2 12:46:32 phase-3
12:47:12 phase-4 12:47:52 phase-5 12:48:09 implement -> review
```
The brief's explicit one-phase-at-a-time requirement was followed, and it
still took under three minutes. Phase count is not the lever; volume of
mechanical work is.
`awaiting_manual_verification` does not hold the phase open either. All five
phases carried that status and the agent moved on regardless.
### The state-file lever cannot be driven with `restart` alone
The documented procedure is stop, edit, start. The unattended sudo rule grants
`install` and `systemctl restart orchestra-worker`, with no `stop` or `start`.
With `restart` only, the running worker's five-second tick save landed between
the edit and the restart, and the edit was gone. Widening the sudoers line is
the fix.
### One cosmetic note
`Session.HandoffRequestedAt` serializes as `0001-01-01T00:00:00Z` when unset,
because `omitempty` does not omit a zero `time.Time`. The worker parses it back
to a zero value and `watchHandoff` starts the clock, so behaviour is correct.
It is the same marshalling trap the operator console hit.
## Run 18, 2026-08-29: the implement-successor rung, proven
Task `06G4VF5HZW7Q4JBM3TTY7W1Y64` on `c587f2c`, a five-phase plan whose phases
are ten named checks rather than one refactor. Volume of mechanical work is
what keeps the implement phase open; phase count does not.
### The rotation
```text
13:43:10 implement session launched
13:43:44 phase-1 verified
13:44:04 phase-2 verified
13:44:35 handoff requested, reason=milestone, delivered by watchHandoff's resend
13:45:25 TaskLeaseRenewed
13:46:51 TaskReleased handoff_ref 0b4fae6f705a anchor 8ea3c7d1ae8f
13:46:51 TaskLeased successor, same harness
13:46:55 TaskPickupValidated
13:47:06 TaskLaunchAcknowledged, still in implement
```
The lever was the worker state file, backdated past `handoffRetryAfter` so the
request went out on the next tick. Under `c587f2c` that lever now produces a
real prompt: the flag alone used to sit there unasked, which is how run 16
ended in a silent expiry.
### What the successor was given
From `.orchestra/launch.md`, written at 13:47:05:
```text
## Current phase
implement: Implement the accepted plan below. Verify as you go.
## Verified git state
- head: 8ea3c7d1ae8fd960897344694645117adbd8ee82
- uncommitted changes: false
## Current human decisions
None recorded. Work from the goal and acceptance above.
## Plan progress
Orchestra established this by running the plan's own verification. You cannot
write it.
- phase-1 (...): automated checks passed at d3acd4989a20, stale because the
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
- phase-2 (...): automated checks passed at d3acd4989a20, stale because the
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
- phase-3 (Checks 7 to 8): not started
- phase-4 (...): not started
- phase-5 (...): not started
```
Everything the rung asked for is there: the complete accepted plan with all
five phases, their files, changes and verification; the phases already
verified; the first unfinished phase; and the current human authority. The
launch context also carries the accepted research and its dead ends.
**SHA staleness renders itself.** Neither phase is reported as simply passed.
Each says the checks passed at `d3acd4989a20` and are stale because the tree
has moved to `8ea3c7d1ae8f`. That is half of the manual-verification rung
observed without being asked for.
### The lever still loses a race, sometimes
`systemctl restart` alone leaves a five-second window in which the running
worker's tick save can clobber the edit. It clobbered run 17's attempt and
survived run 18's. Stop, edit, start is the reliable sequence, and it needs
`systemctl stop` and `start` in the unattended sudo rule.
## Run 19, 2026-08-29: the manual verification seam, and F63
Two tasks, both ingested from Gitea so that comments reconcile:
`06G4W2TMZ9MW9SEBM5MGTM5288` (issue 24, plan `500464b9`) and
`06G4W63545T3RV8SSKGKN11B3G` (issue 26, plan `53f7918b`).
A burn-in task cannot test this rung. Comments reconcile only for the source
the task came from, so a task created with `source: "burnin"` has no comment
stream at all.
### Proven: the keyed seam
```text
15:09:15 PlanPhaseVerified phase-1 awaiting_manual_verification at 80728967
comment: "Looks good to me. Nice work on this one, ship it."
17:10:03 HumanDecisionRecorded subject=operator_instruction
phase-1 unchanged, still awaiting_manual_verification
comment: "orchestra verify phase-1"
17:10:25 HumanDecisionRecorded subject=plan_phase_verification:500464b9…:phase-1
phase-1 verified
```
All five phases reached `awaiting_manual_verification` on their automated
checks alone. A generic approval satisfied nothing. The keyed form flipped
exactly the phase it named, under the plan ref Orchestra supplied rather than
one the commenter chose.
### Proven: re-verification at a new SHA
On issue 26, inside one live implement lease, with the worker stopped for the
whole edit:
```text
15:23:00 PlanPhaseVerified phase-1 awaiting_manual_verification at f5d8f90b
operator commit moves the tree f5d8f90b -> 1f07b89c
.orchestra/plan-progress.json requests phase-1 again
15:23:17 PlanPhaseVerified phase-1 awaiting_manual_verification at 1f07b89c
17:23:16 HumanDecisionRecorded subject=plan_phase_verification:53f7918b…:phase-1
phase-1 verified at 1f07b89c
```
The rerun ran the plan's own commands at the new tree and wrote a fresh record
bound to it. The worker never reads a command out of the request: the request
names a phase and nothing else about it is trusted.
### F63: a manual sign-off outlives the tree it was given against
The second half of the same rig, with no new comment:
```text
operator commit moves the tree 1f07b89c -> 28af84ff
.orchestra/plan-progress.json requests phase-1 again
15:24:02 PlanPhaseVerified phase-1 verified at 28af84ff
```
`RecordPlanPhaseVerification` consults `manuallySignedOff`
(`internal/operations/planprogress.go:127,152`), which asks only whether a
decision with that subject exists. It does, forever, for that plan ref and
phase id. So the automated half of a phase is re-established at every new SHA
while the manual half is asserted from a tree the human never saw.
This is the same class of bug `AtSHA` exists to prevent, applied to the half
that `AtSHA` does not cover. A manual check on this project is a human reading
printed output; an edit between the sign-off and the rerun can change exactly
that output.
Only a replan clears it, because `PlanPhases()` drops records whose plan ref is
not the accepted one.
Proposed shape, not built: when the reducer flips a record on a keyed decision
(`internal/store/store.go:273`), it knows the record it satisfied and that
record's `AtSHA`. Stamp that sha on the decision, and have `manuallySignedOff`
require it to match the run being recorded. A sign-off that arrives before any
run has no sha and keeps today's behaviour, so the ordinary ordering is
unchanged.
### Also established
- `review -> implement` is legal in `legalPhaseTransitions`, but
`AdvanceWorkPhase` only walks the project's path forward. A task that has
left implement can return only through a review that requests changes, not
through the operator phase endpoint.
- The agent's re-verification request is a file an operator can write:
`.orchestra/plan-progress.json`, `{"phase": "...", "status":
"ready_for_verification"}`. It is the whole lever for this rung.
- Ingest turns every issue label into a required capability, so a rig issue
must carry no labels or it will not schedule.
## Run 20, 2026-08-29: the mismatch and replan ladder, and three defects
Task `06G4WJ9T4F35NZC4Z8QQXM9Z6G` (issue 29) on `de18f37`, plus a first attempt
on `06G4WDJ06G7FK00QF73Z9DDBT4` (issue 27) on `c587f2c` that found F65.
The lever throughout is the agent's own file, written by the operator with the
worker stopped: `.orchestra/plan-mismatch.json`, carrying `phase_id`,
`observed`, `contradicts`, `evidence` and `requested_action`. The worker fills
`plan_ref` and `at_sha` from what it can verify, never from the file.
### F65: a task that stops for a human never stopped
Found on the first attempt, live:
```text
16:04:25 PlanMismatchRecorded phase-2 human_decision
16:04:25 worker: plan mismatch refused: task version conflict
16:04:48 PlanPhaseVerified phase-1 work continues as if nothing happened
```
`Store.Append` fences every lifecycle event on a leased task against the
current owner and epoch. `blockForPlanMismatch` carried neither `harness_id`
nor `lease_epoch`, so the block was refused on every task that was actually
running, which is every task that can produce a mismatch. The observation was
durable and the transition silently did not happen: the repo's own documented
failure shape. `raiseTrajectoryGate` had the identical omission, so the
trajectory gate could never have stopped a live task either.
`RecordPlanMismatch` also returns that error after the record is already
appended, so the worker told the agent its report was refused for a report
Orchestra had accepted.
Fixed in `de18f37` with one `fenceToLease` helper, called by all three stops.
The human-decision path already did it correctly and explained why in a
comment; that comment is now the helper.
**Why no test caught it.** `planWith` never leased its task, so every
plan-progress and plan-mismatch test ran in a state no agent can be in. The
`lease` helper in the same package carries a comment warning about exactly
that. It leases now, and the mismatch block test fails without the fence with
the same `task version conflict`.
### Proven: human_decision stops, and the reply resumes the same plan
```text
16:24:21 PlanMismatchRecorded phase-2 human_decision
16:24:21 TaskBlocked plan_mismatch, lifecycle awaiting_human
18:24:46 HumanDecisionRecorded operator_instruction (Gitea's timestamp)
16:25:16 TaskCorrected resumed
16:25:16 TaskLeased plan_ref unchanged at 0bc8ef84
16:25:21 TaskPickupValidated
```
55 seconds from block to resumed, on the same plan, with no reseal. The
blocker packet states the observation, the plan's claim and the evidence.
This is also the live proof of F64: before `6cb2f93` nothing read a blocked
task's comments, because the reconciler ran only before a lease and at a turn
boundary, and a blocked task is never leased.
### Proven: replan keeps the old plan until a replacement seals
```text
16:26:07 PlanMismatchRecorded phase-3 replan
16:26:07 WorkPhaseChanged plan reopen=PlanMismatchRecorded
during planning: plan_ref 0bc8ef84 accepted, phase-1 and phase-2 counted
after the seal:
plan_ref 922f293b
plan_history [0bc8ef84]
progress empty
successor in implement: four phases, all "not started"
```
The successor's launch context carries the replacement whole, with files,
changes and verification per phase. The old progress stops counting because
`PlanPhases()` drops records whose plan ref is not the accepted one. An
abandoned replan would have cost nothing.
### F66: the planner replans blind
The planning session convened by a replan is given the ordinary plan brief and
an empty template. Its launch context says nothing about the plan that already
exists, the contradiction that reopened the phase, the phase id, the evidence,
or the fact that its output supersedes an accepted plan. The only human input
it carried was an unrelated earlier correction.
Everything it needs is already durable. `PlanMismatchRecorded` holds the phase,
the observation, the contradiction and the evidence, and `reopenPhase` writes
`reopen_phase_id` onto the `WorkPhaseChanged` payload. None of it reaches the
agent, so nothing stops the replacement from sealing with the same
contradiction in it.
### Smaller observation
The implement session kept working after the reopen. It verified phase-2
against the old plan 40 seconds after the task had moved to `plan`. Harmless,
because supersession discards it, but the reopen does not end the session that
reported the contradiction. That session rotates at its own next boundary.
## Run 21, 2026-08-29: F66 proven, in two rounds
Tasks `06G4WW6TND26M16CZA6WE5T458` (issue 32, on `edf0076`) and
`06G4XAFH1MBPC35VSJN7V3NS14` (issue 34, on `7539696`).
### Round one failed, and the failure was in the fix
`edf0076` projected the contradiction onto the task and rendered it into the
plan-phase context. The projection appeared at the reopen and was gone before
the planning session launched:
```text
16:58:01 PlanMismatchRecorded phase-2 replan, plan reopened
projection carries the contradiction
16:58:20 TaskReleased the rotation the reopen causes
projection empty
16:59:29 replacement sealed, planner never told anything
```
The clearing rule added with F67 sits **below** the reducer switch, so it runs
for every event rather than for the correction it was written for. A release
found the task unblocked and erased the contradiction. Two mistakes made it:
the rule was written as if it were inside `case "TaskCorrected"`, which is
merely the nearest case above it, and the unit test read the projection at the
moment it was written rather than at the moment the planner reads it.
`7539696` scopes the clear to `TaskCorrected`, and the store test now walks the
real sequence: mismatch, reopen, release, lease, then assert.
### Round two: the planner is told what it was convened to fix
```text
18:00:17 PlanMismatchRecorded phase-2 replan, plan reopened
projection survives the rotation
18:01:02 planning session launches
```
From that session's `.orchestra/launch.md`:
```text
## Why this phase reopened
A plan was already accepted and the code contradicted it. Orchestra reopened
this phase to settle that, and the session that found it is gone.
- phase: phase-2
- observed: The summary lines are printed from the recorded results after
every check has run, ...
- the plan says: The plan states that phase 2 prints the header from a helper
that returns the number of checks about to run.
- evidence: scripts/orchestra_e2e_healthcheck.sh:1
The accepted plan stays accepted until you seal a replacement, and sealing one
supersedes it along with every phase it had verified. Address the
contradiction above: a replacement that repeats it will be contradicted again.
```
It renders above the sealed research and plan, because it changes how they
should be read. The first planning session of a task carries no such section,
and neither does implement. Sealing the replacement (`2edcbe9e`) cleared the
projection, which is the other end of its lifetime.
### The lesson worth keeping
Both rounds of this fix passed their unit tests. What separated them was where
the assertion sat in the sequence. A projection written correctly and read one
rotation later is not the same claim, and only the live run put the read where
the agent does.
+8
View File
@@ -66,6 +66,14 @@ the live herdr instance and check.
`docker logs orchestra-api`. **Deploying a code change means rebuilding the
compose images** (`up -d --build`) — the running image can silently predate
recent commits, so compare its build time against `git log`.
- **A coordinator deploy does not deploy the console.** They are two images
built from the same repo, and the usual `up -d --no-deps orchestra-api`
leaves `orchestra-web-ui` on whatever it was. This has bitten twice: the
ethos console landed in the repo on 2026-08-29 02:44 and was still serving a
2026-07-30 image hours later, through two coordinator deploys. Rebuild it
explicitly from the same clean worktree (`docker build` in `web/`, then
`up -d --no-deps orchestra-web-ui`), and check the served bundle rather than
the commit: `curl -s http://127.0.0.1:19145/assets/<css> | grep 8F7AE5`.
- `orchestra.service` was the **previous** deployment; the unit file was
deleted from `deploy/` on 2026-07-31 along with `redeploy.sh` (which
`sudo install`ed to `/usr/local/bin` and restarted it). If a stale copy is
+110
View File
@@ -408,3 +408,113 @@ which is the priority function slice one deliberately omitted.
None of these required a schema commitment to discover. That was the point of
making the first slice read-only.
## The invariant slice B was written against
**A projection must never manufacture provenance to make evidence easier to
classify.** `task=None` is better than a confident lie.
Both defects the first live run of durable observations exposed were failures
of exactly this rule, and both looked perfectly reasonable in code:
- **False attribution.** The worker's ring outlives the work it describes, so
binding its entries to whatever task the worker is running now produced a
clean, well-formed, wrong association between an old failure and an unrelated
task. The task is read from the observation itself, and only an observation
that names no task belongs to the current lease.
- **Manufactured recurrence.** Treating "still present in the ring" as "it
happened again" turned one failure into four incidents. Presence is not
occurrence: an incident opens only when the entry advances past what has
already been accounted for, and that high-water mark survives the close.
Either one would have corrupted the ledger while every number in it stayed
plausible, which is the specific way this repo's bugs have always presented.
## Signatures are frozen at write time
The signature is computed when an incident is recorded and stored in the event.
Re-projecting an old log with a newer normalizer therefore changes nothing:
running one history through the build before and after the pane-name fix
produced identical output.
That is the intended behavior. An event must not silently change meaning
because normalization code changed. Repairing historical signatures is an
explicit migration or reclassification event, never a different projection
result from the same log.
## What counts as a manual intervention
An intervention is an operator action required to recover, repair, unblock or
correct behaviour that should otherwise have proceeded autonomously. The
ledger measures what the system costs to keep running, so routine operation
does not belong in it.
Counts:
```text
transaction_cleanup
forced_release
state_repair
manual_requeue when recovery failed and a human had to requeue
manual_phase_recovery
worker_restart only when restarting is itself the repair
```
Does not count:
```text
deploy restart
planned upgrade
configuration rollout
normal shutdown and start
deliberate burn-in setup
```
The distinction is policy rather than schema. `worker_restart` is the one kind
that spans both sides, and it stays a single kind until someone actually
misuses it; a field added before the confusion exists is a guess about how it
will be misread.
The first live consequence: the deploy restart of `79d2053` was not recorded,
and `manual_intervention` stayed in the gap list afterwards. That is the
correct result. Orchestra can record the evidence and this history contains
none, which is an honest gap rather than synthetic evidence.
## Slice B status, 2026-08-30
```text
worker observation durability proven live
incident open/close semantics proven live
high-water, no phantom recurrence proven live
task attribution proven live
frozen historical provenance proven live
eligibility calculation proven against real data
eligibility transition unexercised
manual intervention recording implemented
manual intervention provenance unexercised
```
Neither unexercised branch will be manufactured. A second task hitting the
same failure exercises the first, and a genuine repair exercises the second.
The proof that matters from the live run:
```text
recurrence 5, intensity 10, distinct tasks 1, interventions 0 -> eligible=false
```
A noisy failure inside one task does not become system-level maintenance debt.
The three counts stay separate on purpose: recurrence is how often the incident
happened, intensity is how repetitive each incident became, and breadth is how
many independent tasks paid for it. Operational debt requires breadth unless an
operator had to intervene.
### The reasons check is part of the design, not a formality
The aggregate numbers moved in a way that looked like a promotion, and reading
them that way was wrong: the eligible count rose because of an unrelated item.
The pure eligibility explanation contradicted that inference with counted
facts. That is the argument for the ledger resting on mechanically derived
evidence rather than on anyone's reading of what looks recurring, including an
agent's.
+349
View File
@@ -0,0 +1,349 @@
# Handoff: the release path is settled, and the UI became a truth detector
Written 2026-08-29, 14:20 local (10:20 UTC). Read with `BURNIN.md` (the run
ledger, current through run 16), `DEBT-DESIGN.md`, `PLAN-SPEC-DESIGN.md`,
`AUDIT.md` and `CLAUDE.md`. The previous handoff is
`HANDOFF-2026-08-28-plan-v1.md`.
Everything below was observed live unless it says otherwise.
## The headline
**The expired-release defect is fixed and proven both ways.** The whole family
around it is closed. F57 through F60 settle what happens to a release
transaction in every case. That includes the ones that used to need an operator
with a text editor.
**Three new things exist that did not before.** A bounded observation ring on
worker health, closing F18. A read-only debt ledger projected from the event
log. An operator console rebuilt on the ethos design system.
**The UI turned out to be a truth detector.** Nine screens were built against
real endpoints. They found four places where Orchestra has no capability to
support the intended interface. That list is the most valuable output of the
session.
```text
orchestra-plan-v1 plan machinery proven
orchestra-f18-baseline d6ee10f, the bounded observation ring
↓ 6 commits
34f3c28 deployed now: release path settled, debt ledger, new UI
```
## Deployed state
| Half | Revision |
|---|---|
| Coordinator, homesrv container | `34f3c28` |
| Worker, workpc systemd | `34f3c28` |
```text
commit 34f3c2888fc7d45d190d93e1ea42501b6cd3e474
coordinator sha256 1d32d83ec859b36e12473fab01bbfc3c769b97b4f6ae4db2841db5161e0eda19
worker sha256 0e3877321dea8a1eeda51ccc6f3ead1a14aa5f5cae4d95b704f61c364248ec65
```
`cbd6b11` is one commit above and is documentation only. Three commits are
unpushed.
**Worker installs no longer need a human.** The operator installed the
`/etc/sudoers.d` line, so `sudo -n install …` and
`sudo -n systemctl restart orchestra-worker` both work unattended. Verify the
running revision from the journal, never the installed file.
## The defects fixed, and how each was found
Not one came from reading code. Every one came from a live run failing.
| Id | Commit | What |
|---|---|---|
| F57 | `6565b9f` | An expired lease could never commit the anchor it had already pushed. The worker sent an epoch the expiry replay had deleted, and the coordinator refused any handoff without a live lease. The epoch now belongs to the transaction, `TaskReleased` retains the ending epoch, and `lateHandoffAccepted` lets exactly that owner commit while the task is queued and unleased. |
| F58 | `03663f4` | A superseded transaction retried a permanent 409 forever, holding the pane and pinning `ActiveTask`. Run 10's task did it for seven hours. `TaskLeased` now abandons a transaction whose id the lease does not carry. |
| F59 | `8e37989` | F58 fires on `TaskLeased`, and a failed task is never leased again. `TaskFailed` now drops the transaction too. |
| F60 | `3c7cf95` | The general rule the other two were reaching for. Terminal is failed or completed. Blocked keeps the transaction, because a reopen can still commit it, so `TaskBlocked` now retains the ending epoch as well. A refusal parks the commit for 30s backing off to 5 minutes, and any event about the task un-parks it. A transport failure is not an answer and retries at once. |
| F18 | `d6ee10f` | The single `last_error` slot. Worker health now carries up to sixteen distinct observations with repeat counts and first/last times, collapsing by message rather than by position. |
### The rig that proved F57, and the guard
```text
19:00:10.742 transaction opens at prepared, anchor pushing
19:00:10.727 TaskReleased v15 reason=lease_expired surface=tui
19:00:11.662 TaskReleased v16 the late commit, accepted 935ms after the lease died
19:00:11.665 TaskLeased v17 successor picks up the handoff
19:00:15.524 TaskPickupValidated v18
```
Race guard, next boundary: force the expiry, then lease the task to a probe
harness before the push finishes. The late commit is refused, no handoff is
written, and the successor's lease stands.
**The rig technique matters more than the rig.** Suspending the worker cannot
produce this ordering. The event replay runs at the top of every tick and
discards the transaction. The ordering exists only inside one call:
transaction opened, anchor pushing, commit not yet sent. So poll the worker
state file at 2ms and fire `POST /v1/tasks/<id>/release` the instant a
transaction appears at `prepared`.
**A named probe harness owns a lease without starting an agent.**
`race-guard-probe` never picks anything up and expires on the normal TTL.
## Corrections to the previous handoff
**The operator lifecycle actions do not lose a version race.** On a leased
task, `block`, `release` and `attention` are refused by
`internal/store/store.go:885-901` when the payload omits `harness_id` and
`lease_epoch`. Ten attempts in 550ms all failed that way. Send both fencing
fields and they succeed on the first try.
**The OpenCode Zen free tier is not blocked.** The selected model was.
## The debt ledger
`DEBT-DESIGN.md` answers nine design questions and carries four amendments the
operator made. Slice one is built, deployed and run.
**Slice one writes nothing.** A read-only projection over the existing log,
plus a pure eligibility function and `GET /v1/debt`. It folded 881 events and
produced 15 candidates and 3 gaps.
```text
v1:operational:lease_expired:workpc-opencode:lease r=41 tasks=4
v1:operational:lease_expired:workpc-claude:lease r=29 tasks=13
v1:operational:lease_failure:-:lease r=20 tasks=16
v1:correctness:handoff_validation:-:lease r=5 tasks=5
```
The opencode failure shape is the top item, found mechanically. Run 14 reached
the same conclusion by hand from a pane capture.
**It reported what it cannot see, which was the point.** The 409 release loop
does not appear. That evidence lived in the F18 worker ring, and no event
carries it. Manual interventions are a non-durable gap for the same reason.
**The first run exposed four defects in the model**, all recorded at the end of
`DEBT-DESIGN.md`:
- the component part is too coarse for lease evidence
- `harness` is often empty on block reasons
- path normalization mangled a mismatch reference
- recurrence alone is the wrong sort order
Layout, so the read model does not end up in the command layer:
```text
internal/domain/debt.go types, signature, classification
internal/store/debt_projection.go the fold, and the gap report
internal/operations/debt.go CheckDebtEligibility
```
## The operator console
Nine screens on the ethos system, signal violet `#8F7AE5`, routing fork motif.
Each screen was built by its own agent against a foundation with one author.
The shell, tokens and primitives could not drift into nine dialects.
**Render before signing off.** Three bugs existed that no computed value would
have caught. All three came from looking at a screenshot:
- The previous stylesheet fought every shared class name and leaked properties
the new rules never mention, which is how `position: fixed` survived on
`.topbar`. It is now scoped under `.legacy` and reaches only the login route.
That also stops its green accent and its `backdrop-filter` from reaching the
console.
- Go marshals a zero `time.Time` as `0001-01-01T00:00:00Z` and `omitempty` does
not omit a struct, so absent timestamps arrived populated-looking and
rendered as `739855d ago`. Stripped once in `client.ts`, with a test.
- Long machine ids overflowed their cards and painted under the next one.
Chromium is installed at `/usr/bin/chromium`. To see a screen without a live
session, write a throwaway harness that stubs `window.fetch` and renders
`<Console>` inside a `MemoryRouter`, served by vite on a spare port. Note that
`npx` and `./node_modules/.bin/*` do not work on this filesystem: call
`node ./node_modules/vite/bin/vite.js` directly.
## What the UI proved Orchestra cannot do
This is the part worth acting on. Each screen refused to fake something, and
the refusals name real capability gaps.
| Gap | Evidence |
|---|---|
| **No web-facing human-decision write path** | `Steer / Correct` is disabled. `internal/ui/ui.go`'s action switch has grant/deny approval, resubmit, handoff, release, block and complete, and nothing writes `HumanDecisionRecorded`. The spec makes steering the primary action of the task detail screen. |
| **No keystroke forwarding** | `Take control` is disabled. Only resubmit and approval grant/deny reach a live pane. |
| **Context occupancy is trapped in herdr** | Three screens independently hit it. No projection carries it. |
| **Project configuration is not served** | Repo, remote, quality gate and verification policy live only in `config.jsonc`. The projects screen can show none of it. |
| **The web cannot request a handoff for a federated task** | `RequestHandoff` needs a local coordinator and answers 503. That is the Design A guardrail working. |
The operator's direction on these. Treat first-class direct human input as the
highest-value backend feature. Build it as `POST /v1/tasks/<id>/decisions`,
using the same durable decision semantics as Gitea comments, so Gitea, CLI and
web converge on one `HumanDecisionRecorded`. Keep take-control disabled,
because arbitrary pane input bypasses the durable authority model. Expose
occupancy through a session health projection rather than teaching the web
server about herdr. Add a read-only effective project configuration endpoint,
which F61 will also need.
## Where the plan-machinery ladder stands
Proven in run 14: the worker executes the **sealed plan's** commands rather
than the request's, and every `PlanPhaseVerified` binds `plan_ref`, `phase_id`,
`at_sha`, `evidence_ref`, `lease_epoch` and `harness_id`.
Proven in run 16: a successor inherits the **whole sealed plan**, all phases
with their files, changes, verification, and the research citations.
**Still unproven, and the next runtime item:**
```text
mid-implement rotation
→ successor picks up in implement
→ launch context states which phases are already verified
manual verification
→ SHA goes stale
→ re-verification
plan mismatch
→ human decision
→ real replan, old plan retained, replacement launched
```
Two things make the first one hard, and both are now known:
- **A trivial task spends about four minutes in implement**, and verifies every
phase against one tree near the end. The implementer writes the whole change
first, then verifies each phase in turn. Use a task whose implement phase
genuinely runs long.
- **A state-file edit does not survive a running worker.** It holds sessions in
memory and writes them back. Suspend, edit, then `sudo systemctl restart`.
Resuming lets the old copy win. Setting `handoff_requested` on the session is
the production rotation lever.
Progress is withheld from a **review** successor on purpose
(`internal/agentctx/agentctx.go:321`), because an independent review must
reconstruct the change from the diff. That absence is not the defect.
## F61 and F62, recorded and not built
**F61: the planner learns the verification policy by refusal.** The brief says
a policy exists, not what is in it. Every plan therefore pays one refused round
trip. The recovery loop works: run 14's planner consumed the refusal and
resealed 31 seconds later. It is not a one-liner, because `agentctx.Input.Policy` is filled
from the worker's `SafeOperations` while the verification policy is
coordinator-side. The reason to promote it later is local models, which may
propose forbidden commands repeatedly because they cannot infer the allowed
substitute.
**F62: a requested handoff nobody answers is invisible.** The rotation is
agent-driven. In run 16 the agent never wrote `HANDOFF.md`, renewals stopped,
the lease expired, and the task lost an attempt. Worker health recorded only
`agent status idle and pane unchanged`, 34 times. There is no timeout, no
retry, and no observation saying a handoff was requested and left unanswered.
The expiry is indistinguishable from an ordinary idle one.
## Live state
Three pull requests are open and unreviewed:
```text
06G4M6HF1Z3EREX1X3NEKSHP24 pulls/18
06G4M8WHGQ4P3GQMPEEH0RJRHM pulls/19
06G4SWEVP71FYYKAV5FV0ZK5ZG pulls/20
```
Both workers are online on `34f3c28` with no release transactions and no
sessions. 29 blocked `test-e2e` tasks are burn-in debris. Three `correx` tasks
are queued and unschedulable, because `correx` has no entry in the
coordinator's `config.jsonc`.
**opencode now runs.** The model was the problem, and `hy3-free` works. It then
stops on a permission prompt, because `~/.config/opencode/opencode.jsonc` sets
`"bash": "ask"`. Orchestra can answer that exact dialog, but only when an
operator queues `grant_approval`, so unattended work stalls on the first
command. Set `"bash": "allow"` for unattended runs. That file also has no
top-level `model` key, so OpenCode picks whatever sits at the top of
`~/.local/state/opencode/model.json`, which any manual pick silently changes.
**The opencode adapter is now a debt item, not a curiosity.** It cannot resolve
a session file. Activity therefore reads `unknown`, and the worker cannot tell
finished from never-started. The debt projection surfaced it independently as
the top recurring operational item.
## Things that will bite
- **Background python tasks get killed here.** Three watchers died before doing
anything. Foreground polling and the `Monitor` tool both work.
- **`/v1/events` is one line of JSON.** A `grep` for two substrings matches
across unrelated tasks. Parse it.
- **`npx` and `./node_modules/.bin/*` fail on this filesystem.** Call node
directly.
- **`rm` and `cp` are interactive.** Use `/bin/rm -f` and `install`.
- **Secrets are guarded.** Expand a token inside the container in one remote
command:
`T=$(docker exec orchestra-api printenv ORCHESTRA_TUI_TOKEN); curl -s -H "X-Orchestra-Surface: tui" -H "Authorization: Bearer $T" ...`
- **Rebuild the coordinator from a detached worktree**, and split the worktree
add, the docker build and the compose up into separate commands.
- **`deploy/build.sh` builds both halves from one commit with one stamp.** Use
it.
- **Geist is not on disk.** Both stacks fall back to system faces, and the
ethos threat model rules out the font CDN.
## Resume in this order
Set by the operator at the session boundary. Do not rebuild it from the commits.
1. **F62 first.** Make a requested-but-unanswered handoff visible and bounded.
Preserve the lease while Orchestra is explicitly waiting, retry the confirmed
request, then emit a causal timeout. Today it ends as generic idleness.
2. **Repeat the implement-successor rung** with a deliberately longer task. Get
at least one `PlanPhaseVerified`, force the handoff while still in
`implement`, and prove the successor sees the complete accepted plan, the
verified previous phase, the first unfinished phase, and the current human
authority.
3. **Finish the remaining ladder.** Manual verification, SHA staleness and
reverification, human-decision mismatch, then a real replan with old-plan
provenance.
4. **Use the capability table above as backend work discovery.** Do not add fake
controls. Each disabled action is concrete evidence of a missing capability.
5. **Continue the debt slices independently.** Durable worker observations and
manual-intervention events are the next evidence gaps. Not automatic
maintenance yet.
## What the tag means
```text
orchestra-release-v1 -> 34f3c28
deployed, settled lifecycle plus the truth-detector UI baseline
later HEADs
experimental plan, debt and runtime work that must earn their own
release proof
```
Use that distinction. A defect found in experimental work is not a reason to
reopen settled architecture.
## The roadmap the operator set
```text
A. runtime correctness finish the plan-machinery live proof
B. evidence and debt durable worker observations, manual-intervention
events, then rerun the projection
C. operator surface first-class HumanDecision write API,
effective project-config read API,
session and context health projection
D. adapter fix opencode activity and session resolution
E. UI wire capabilities as backend support becomes real
```
The framing that ties them together is the thing to keep. The UI says what
Orchestra cannot expose or control. The debt ledger says which of those
shortcomings repeatedly costs something. The burn-in says which runtime
semantics are reliable. Those three decide what gets built next.
One design question to settle before B's slice two writes any code. The worker
ring is bounded and lossy by construction. Ingesting it durably means deciding
whether the coordinator stores every observation as an event, or only
transitions. Storing every heartbeat's ring would write the same 41-count
observation hundreds of times.
+17
View File
@@ -0,0 +1,17 @@
# orchestra ui mockups
`final/` contains the nine accepted/current screen directions:
1. dashboard
2. task detail
3. terminal live pane
4. tasks
5. decisions
6. workers
7. projects
8. review
9. settings
`iterations/` contains every generated mockup from the design session, including superseded variants.
`orchestra-ui-spec.md` is the accompanying implementation/design specification.
+56
View File
@@ -0,0 +1,56 @@
// Runs the debt projection over a captured event log and prints one row per
// item, so two builds can be compared on identical input. The point of the
// comparison is the signature fix: cross-task incidents must collapse into one
// item while genuinely different failures stay separate.
package main
import (
"encoding/json"
"fmt"
"os"
"sort"
"orchestra/internal/domain"
"orchestra/internal/operations"
"orchestra/internal/store"
)
func main() {
b, err := os.ReadFile(os.Args[1])
if err != nil {
panic(err)
}
var events []domain.Event
if err := json.Unmarshal(b, &events); err != nil {
panic(err)
}
ledger := store.ProjectDebt(events)
eligible := map[string]operations.DebtCandidate{}
for _, c := range operations.EligibleDebt(ledger) {
eligible[c.Item.ID] = c
}
rows := make([]map[string]any, 0, len(ledger.Items))
for _, item := range ledger.Items {
tasks := map[string]bool{}
intensity := 0
for _, o := range item.Observations {
if o.TaskID != "" {
tasks[o.TaskID] = true
}
intensity += o.Repeats
}
row := map[string]any{
"signature": item.ID, "class": string(item.Class),
"recurrence": len(item.Observations), "intensity": intensity,
"tasks": len(tasks), "eligible": false, "reasons": []string{},
}
check := operations.CheckDebtEligibility(item)
row["eligible"] = check.Eligible
row["reasons"] = check.Reasons
_ = eligible
rows = append(rows, row)
}
sort.Slice(rows, func(i, j int) bool { return rows[i]["signature"].(string) < rows[j]["signature"].(string) })
out, _ := json.MarshalIndent(map[string]any{"items": rows, "gaps": ledger.Gaps}, "", " ")
fmt.Println(string(out))
}
+110
View File
@@ -0,0 +1,110 @@
package main
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"orchestra/internal/domain"
"orchestra/internal/federation"
"orchestra/internal/herdr"
)
// F62. Run 16: the agent was asked to hand off, never wrote HANDOFF.md,
// renewals stopped, and the lease died as ordinary idleness. Waiting is now
// bounded: re-ask once, then give the task up with a class that says why.
func TestUnansweredHandoffIsRetriedThenGivenUp(t *testing.T) {
var nack map[string]any
w, backend, _, done := phaseWorker(t, func(rw http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/nack") {
_ = json.NewDecoder(r.Body).Decode(&nack)
}
rw.Write([]byte(`{}`))
})
defer done()
ctx := context.Background()
requested := func(ago time.Duration) herdr.Session {
s := w.sessions["task"]
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
s.HandoffRequestedAt = time.Now().UTC().Add(-ago)
w.sessions["task"] = s
return s
}
// Still inside the answering window: nothing said, nothing given up.
if s, gaveUp := w.watchHandoff(ctx, "task", requested(time.Minute)); gaveUp || s.HandoffRetried {
t.Fatalf("gave up while still waiting: gaveUp=%v session=%+v", gaveUp, s)
}
if len(backend.prompts) != 0 {
t.Fatalf("re-asked too early: %q", backend.prompts)
}
// Past the retry point: asked again, exactly once.
s, gaveUp := w.watchHandoff(ctx, "task", requested(handoffRetryAfter+time.Minute))
if gaveUp || !s.HandoffRetried || len(backend.prompts) != 1 {
t.Fatalf("retry: gaveUp=%v retried=%v prompts=%q", gaveUp, s.HandoffRetried, backend.prompts)
}
if _, gaveUp = w.watchHandoff(ctx, "task", s); gaveUp || len(backend.prompts) != 1 {
t.Fatalf("re-asked every tick: %q", backend.prompts)
}
// Past the bound: a causal reclaim, and no lease left to renew.
s = requested(handoffAnswerTimeout + time.Second)
s.HandoffRetried = true
w.sessions["task"] = s
if _, gaveUp = w.watchHandoff(ctx, "task", s); !gaveUp {
t.Fatal("an unanswered handoff waited forever")
}
if nack["failure_class"] != "handoff_unanswered" {
t.Fatalf("nack = %+v", nack)
}
if detail, _ := nack["last_error"].(string); !strings.Contains(detail, "phase_changed") {
t.Fatalf("the reclaim does not name the request: %q", detail)
}
if _, held := w.leases["task"]; held {
t.Fatal("the given-up task kept its lease")
}
}
// The lease must survive the wait it was asked to make: an idle pane is the
// answer Orchestra requested, not evidence of an agent that stopped working.
func TestWaitingForAHandoffKeepsTheLease(t *testing.T) {
renewals := 0
api := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
renewals++
rw.Write([]byte(`{}`))
}))
defer api.Close()
backend := &recordingBackend{status: "idle", progress: "same screen"}
w := &worker{
api: federation.Client{BaseURL: api.URL, WorkerID: "h", Token: "t"},
backend: backend,
harness: "claude",
sessions: map[string]herdr.Session{"task": {PaneID: "pane", HandoffRequested: true, HandoffRequestedAt: time.Now().UTC()}},
leases: map[string]lease{"task": {Epoch: "e", Version: 1, Until: time.Now(), ProgressSHA: domain.Hash([]byte("same screen"))}},
quarantined: map[string]bool{},
statePath: filepath.Join(t.TempDir(), "state.json"),
}
w.renewLeases(context.Background())
if renewals != 1 {
t.Fatalf("a lease waiting on a requested handoff renewed %d times, want 1", renewals)
}
// Past the bound the exemption stops: watchHandoff has given the task up
// by then, and nothing keeps an unanswered request alive.
s := w.sessions["task"]
s.HandoffRequestedAt = time.Now().UTC().Add(-handoffAnswerTimeout - time.Second)
w.sessions["task"] = s
l := w.leases["task"]
l.Until = time.Now()
w.leases["task"] = l
w.renewLeases(context.Background())
if renewals != 1 {
t.Fatalf("the exemption outlived its bound: renewals=%d", renewals)
}
}
+96 -5
View File
@@ -103,8 +103,13 @@ func (w *worker) recordError(err error) {
w.observations = append(w.observations, federation.Observation{Message: msg, Count: 1, First: now, Last: now})
}
// workerIncarnation identifies this process. A restarted worker cannot
// continue the previous process's failures, and nothing else on the wire says
// a restart happened: build revision and worker id both survive it.
var workerIncarnation = domain.NewID()
func (w *worker) health(ctx context.Context) federation.WorkerHealth {
h := federation.WorkerHealth{HerdrStatus: "unknown"}
h := federation.WorkerHealth{HerdrStatus: "unknown", Incarnation: workerIncarnation}
if backend := w.executionBackend(); backend != nil {
h.Backend = backend.Kind()
}
@@ -174,6 +179,7 @@ func releaseBackoff(attempts int) time.Duration {
}
return d
}
type projectConfig struct {
Repo string `json:"repo"`
Root string `json:"worktree_root"`
@@ -673,6 +679,13 @@ func (w *worker) releaseReady(ctx context.Context) {
w.advanceRelease(ctx, id, s)
continue
}
if s.HandoffRequested {
next, gaveUp := w.watchHandoff(ctx, id, s)
if gaveUp {
continue
}
s = next
}
w.rotationTick(ctx, id, s)
}
}
@@ -772,7 +785,7 @@ func (w *worker) rotationTick(ctx context.Context, id string, s herdr.Session) {
w.recordError(fmt.Errorf("rotation %s threshold prompt: %w", id, err))
return
}
s.HandoffRequested, s.HandoffReason = true, d.Reason
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, d.Reason, time.Now().UTC()
w.sessions[id] = s
_ = w.save()
}
@@ -1166,6 +1179,70 @@ func (w *worker) submit(ctx context.Context, id string, s herdr.Session, e compl
// the directory also holds for a worktree that has no inner .gitignore.
const stageExclude = ":!.orchestra"
// F62: a requested handoff nobody answers was invisible. Renewals stopped,
// the lease expired, and the task lost an attempt with nothing on record
// saying a handoff had ever been asked for — worker health showed only "agent
// status idle and pane unchanged", 34 times in run 16.
const (
handoffRetryAfter = 4 * time.Minute
handoffAnswerTimeout = 10 * time.Minute
)
// watchHandoff bounds the wait for an agent's handoff answer: re-send the
// request once, then give the task up with a class that names the cause. It
// returns the session to keep using and whether the task was given up.
func (w *worker) watchHandoff(ctx context.Context, id string, s herdr.Session) (herdr.Session, bool) {
if s.HandoffRequestedAt.IsZero() {
// A session persisted before the stamp existed, or requested by a path
// that does not set it. Start the clock now rather than time out a
// request retroactively.
s.HandoffRequestedAt = time.Now().UTC()
w.sessions[id] = s
_ = w.save()
return s, false
}
waited := time.Since(s.HandoffRequestedAt)
if waited < handoffRetryAfter {
return s, false
}
if waited < handoffAnswerTimeout {
if s.HandoffRetried || w.executionBackend() == nil {
return s, false
}
a := herdr.CLIAdapter{Backend: w.executionBackend(), Harness: w.harness}
var err error
if s.HandoffReason != "" {
err = a.RequestHandoffReason(ctx, s, s.HandoffReason, nil)
} else {
err = a.RequestHandoff(ctx, s)
}
if err != nil {
w.recordError(fmt.Errorf("handoff %s re-request: %w", id, err))
return s, false
}
s.HandoffRetried = true
w.sessions[id] = s
_ = w.save()
w.recordError(fmt.Errorf("handoff %s (%s) unanswered for %s: request re-sent", id, s.HandoffReason, waited.Round(time.Second)))
return s, false
}
l, ok := w.leases[id]
if !ok {
return s, false
}
detail := fmt.Sprintf("handoff requested (%s) and unanswered for %s", s.HandoffReason, waited.Round(time.Second))
if err := w.api.NackStart(ctx, id, l.Epoch, l.Version, "handoff_unanswered", detail, w.sessionEvidence(ctx, id, s)); err != nil {
w.recordError(fmt.Errorf("handoff timeout %s: %w", id, err))
return s, false
}
// The coordinator answers with TaskReleased; its replay quarantines the
// pane. Drop the lease here so nothing renews it in the meantime.
delete(w.leases, id)
_ = w.save()
w.recordError(errors.New(detail))
return s, true
}
func (w *worker) renewLeases(ctx context.Context) {
if w.executionBackend() == nil {
return
@@ -1199,6 +1276,11 @@ func (w *worker) renewLeases(ctx context.Context) {
case l.ProgressSHA == "":
// First renewal has no baseline to compare against. Record one and
// allow this renewal; the next one must show real movement.
case s.HandoffRequested && !s.HandoffRequestedAt.IsZero() && time.Since(s.HandoffRequestedAt) < handoffAnswerTimeout:
// Orchestra told this agent to stop and write its handoff. A quiet
// pane is the answer it was asked for, so the lease is held while
// the wait is explicitly bounded (F62). Past the bound the case
// stops matching and watchHandoff has already given the task up.
default:
w.recordError(fmt.Errorf("lease %s not renewed: agent status %s and pane unchanged since the last renewal", taskID, status))
continue
@@ -1807,7 +1889,7 @@ func main() {
window: window,
// Capacity stays one per identity because a herdr's declared
// concurrency is one. Serving N harnesses gives the process N slots.
registration: federation.Worker{ID: spec.ID, Address: spec.Address, Capacity: 1, SupportedProjects: supported, Build: buildinfo.Current()},
registration: federation.Worker{ID: spec.ID, Address: spec.Address, Capacity: 1, SupportedProjects: supported, Build: buildinfo.Current(), Incarnation: workerIncarnation},
}
if w.registration.Address == "" {
w.registration.Address = os.Getenv("ORCHESTRA_WORKER_ADDRESS")
@@ -1932,7 +2014,7 @@ func (w *worker) federatedTurn(ctx context.Context, id string, a herdr.Adapter,
w.recordError(fmt.Errorf("reconcile failure handoff %s: %w", id, err))
return
}
s.HandoffRequested, s.HandoffReason = true, "reconcile_failure"
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "reconcile_failure", time.Now().UTC()
w.sessions[id] = s
_ = w.save()
return
@@ -2009,9 +2091,18 @@ func (w *worker) rotateForPhase(ctx context.Context, id string, a herdr.Adapter,
w.recordError(fmt.Errorf("phase rotation %s: %w", id, err))
return
}
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "phase_changed", time.Now().UTC()
w.sessions[id] = s
_ = w.save()
// A request belongs to the session that wrote it. Both files sit in the
// worktree, which outlives the session, so a successor in a different
// phase would find and execute them: that is how a reopened planning
// session verified a phase of the plan it was replacing.
for _, name := range []string{planProgressFile, phaseRequestFile} {
if err := os.Remove(filepath.Join(s.Worktree, ".orchestra", name)); err != nil && !os.IsNotExist(err) {
w.recordError(fmt.Errorf("phase rotation %s: drop %s: %w", id, name, err))
}
}
log.Printf("phase changed for %s: session rotating", id)
}
+29
View File
@@ -457,3 +457,32 @@ func TestPlanVerificationRunsThePlansCommandsAndReportsExitCodes(t *testing.T) {
t.Fatalf("the outcome was not delivered: %v", backend.prompts)
}
}
// A request belongs to the session that wrote it. The worktree outlives the
// session, so a rotation that leaves these files behind hands them to a
// successor running in a different phase (run 20).
func TestRotationDropsTheEndedSessionsRequests(t *testing.T) {
w, backend, wt, done := phaseWorker(t, func(rw http.ResponseWriter, r *http.Request) {
rw.Write([]byte(`{}`))
})
defer done()
progress := filepath.Join(wt, ".orchestra", planProgressFile)
request := filepath.Join(wt, ".orchestra", phaseRequestFile)
for _, p := range []string{progress, request} {
if err := os.WriteFile(p, []byte(`{"phase":"phase-1","status":"ready_for_verification"}`), 0o644); err != nil {
t.Fatal(err)
}
}
a := herdr.CLIAdapter{Backend: backend, Harness: "claude"}
w.rotateForPhase(context.Background(), "task", a, w.sessions["task"])
for _, p := range []string{progress, request} {
if _, err := os.Stat(p); !os.IsNotExist(err) {
t.Fatalf("%s survived the rotation that ended the session that wrote it", filepath.Base(p))
}
}
if s := w.sessions["task"]; !s.HandoffRequested || s.HandoffReason != "phase_changed" {
t.Fatalf("the session was not rotated: %+v", s)
}
}
+94
View File
@@ -228,6 +228,9 @@ func main() {
// Pull-request readers by source name, for reflecting submitted work.
pullRequests := map[string]human.PullRequestSource{}
localMachine := os.Getenv("ORCHESTRA_MACHINE_ID")
// One tracker for the process: it holds the per-incident accumulation that
// makes an evicted and recreated ring entry add up instead of restarting.
observations := &operations.ObservationTracker{Store: s}
workers := &federation.Registry{AdmitToken: os.Getenv("ORCHESTRA_FEDERATION_ADMIT_TOKEN"), StatePath: filepath.Join(dir, "federation-state.json")}
if err := workers.Load(); err != nil {
log.Fatalf("load federation state: %v", err)
@@ -445,6 +448,49 @@ func main() {
}
json.NewEncoder(w).Encode(s.Events(n))
})
mux.HandleFunc("/v1/interventions", func(w http.ResponseWriter, r *http.Request) {
// The one command an operator has for saying "I fixed this by hand".
// Everything Orchestra does to itself is already an event; a manual
// repair is the only kind of recovery that leaves no trace unless the
// person who made it says so.
if r.Method == http.MethodGet {
out := make([]domain.Event, 0)
for _, e := range s.Events(0) {
if e.Type == domain.EventOperatorInterventionRecorded {
out = append(out, e)
}
}
json.NewEncoder(w).Encode(out)
return
}
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
if err := authz.AuthorizeEvent(surface(r), domain.EventOperatorInterventionRecorded); err != nil {
http.Error(w, err.Error(), http.StatusForbidden)
return
}
var in domain.OperatorIntervention
if json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<16)).Decode(&in) != nil {
http.Error(w, "invalid json", http.StatusBadRequest)
return
}
e, err := operations.RecordIntervention(s, surface(r), in)
switch {
case errors.Is(err, domain.ErrNotFound):
http.Error(w, err.Error(), http.StatusNotFound)
return
case errors.Is(err, domain.ErrInvalid):
http.Error(w, err.Error(), http.StatusBadRequest)
return
case err != nil:
http.Error(w, err.Error(), http.StatusConflict)
return
}
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(e)
})
mux.HandleFunc("/v1/debt", func(w http.ResponseWriter, r *http.Request) {
// Read-only, and deliberately so. This projection is evidence about
// history, not a new kind of truth: nothing here writes an event,
@@ -1391,6 +1437,29 @@ func main() {
http.Error(w, err.Error(), 404)
return
}
// The ring carries messages and nothing else, so the coordinator
// attributes them: the worker's active task, and that task's
// current lease epoch, are what bind an incident to the work it
// happened during. A failed fold is logged rather than failing the
// heartbeat, because losing evidence about a worker is not a reason
// to also stop believing the worker is alive.
report := operations.WorkerReport{
WorkerID: parts[3], Incarnation: health.Incarnation,
TaskID: health.ActiveTask, At: time.Now().UTC(),
}
if report.TaskID != "" {
if t, ok := s.Task(report.TaskID); ok && t.Lease != nil {
report.LeaseEpoch = t.Lease.Epoch
}
}
for _, o := range health.Observations {
report.Observations = append(report.Observations, domain.WorkerObservation{
Message: o.Message, Count: o.Count, First: o.First, Last: o.Last,
})
}
if _, err := observations.Ingest(report); err != nil {
log.Printf("observation incidents for %s: %v", parts[3], err)
}
w.WriteHeader(http.StatusNoContent)
return
}
@@ -1589,6 +1658,12 @@ func main() {
case "invalid_handoff":
typ = "TaskBlocked"
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonHandoffValidation), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_nacked", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
case "handoff_unanswered":
// F62: not a launch failure. The agent was asked to hand off
// and never did, so the reclaim says exactly that instead of
// arriving as an ordinary idle expiry.
typ = "TaskReleased"
p, _ = json.Marshal(map[string]any{"reason": "handoff_unanswered", "failure_class": b.FailureClass, "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "handoff_unanswered", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
case "launch_uncertain":
typ = "TaskNeedsAttention"
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonLeaseFailure), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_uncertain", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
@@ -1825,6 +1900,25 @@ func main() {
coordinator.ReconcileFailureHandoff = v
}
}
// The third reconciliation point, and the same reasoning the submitted
// work loop above already uses: a blocked task cannot be leased, so a
// pre-lease hook can never see the reply that should unblock it (F64).
// Orchestra stopped and asked the human a question; nothing else here
// will go and read the answer.
go func() {
ticker := time.NewTicker(time.Minute)
defer ticker.Stop()
for range ticker.C {
for _, t := range s.Tasks() {
if t.State != domain.StateBlocked || !domain.BlockReasonAwaitsReply(t.BlockReason) {
continue
}
if err := reconciler.Reconcile(context.Background(), t.ID); err != nil {
log.Printf("reconcile blocked task %s: %v", t.ID, err)
}
}
}
}()
}
if path := os.Getenv("ORCHESTRA_JSONL"); path != "" {
sup := &provider.Supervisor{Name: "jsonl", Run: func(ctx context.Context) error {
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

+32
View File
@@ -314,6 +314,9 @@ func renderTask(in Input) string {
fmt.Fprintf(&b, "- head: %s\n", fallback(in.Git.HeadSHA))
fmt.Fprintf(&b, "- uncommitted changes: %t\n", in.Git.Dirty)
// Above the sealed artifacts, because it is the reason this session exists
// and it changes how the artifacts below should be read.
b.WriteString(renderReopen(in))
b.WriteString(renderSealed(in))
// Below the plan, above continuity. Progress is a fact about the plan, so
// it follows the plan; continuity is one predecessor's account, so it
@@ -499,6 +502,35 @@ func renderPlanProgress(in Input) string {
return b.String()
}
// renderReopen states the contradiction that reopened this phase. Without it
// the planning session a replan convenes gets the ordinary plan brief and an
// empty template, with nothing saying a plan already exists, what contradicted
// it, or that its output supersedes an accepted plan (F66). Everything here is
// durable on PlanMismatchRecorded; it simply never reached the agent.
//
// Only the phases a mismatch can reopen render it. An implementer that finds
// the next contradiction is looking at the code, not at this account of the
// last one.
func renderReopen(in Input) string {
m := in.Task.PlanMismatch
if m == nil || (in.Phase != domain.WorkPhasePlan && in.Phase != domain.WorkPhaseResearch) {
return ""
}
var b strings.Builder
b.WriteString("\n## Why this phase reopened\n\n")
fmt.Fprintf(&b, "A plan was already accepted and the code contradicted it. Orchestra reopened this phase to settle that, and the session that found it is gone.\n\n")
fmt.Fprintf(&b, "- phase: %s\n", collapse(m.PhaseID))
fmt.Fprintf(&b, "- observed: %s\n", collapse(m.Observed))
fmt.Fprintf(&b, "- the plan says: %s\n", collapse(m.Contradicts))
for _, e := range m.Evidence {
fmt.Fprintf(&b, "- evidence: %s\n", collapse(e))
}
if in.Phase == domain.WorkPhasePlan {
b.WriteString("\nThe accepted plan stays accepted until you seal a replacement, and sealing one supersedes it along with every phase it had verified. Address the contradiction above: a replacement that repeats it will be contradicted again.\n")
}
return b.String()
}
func renderSealed(in Input) string {
var b strings.Builder
research := in.Research
+58
View File
@@ -734,3 +734,61 @@ func TestPendingManualGateRendersStale(t *testing.T) {
t.Errorf("a pending manual gate at HEAD rendered stale:\n%s", fresh)
}
}
// F66, found live on run 20. A replan convenes a planning session to settle a
// contradiction, and that session was told nothing about it: the ordinary plan
// brief, an empty template, and no mention that a plan already exists.
func TestAReopenedPlanPhaseIsToldWhatContradictedThePlan(t *testing.T) {
in := input()
in.Phase = domain.WorkPhasePlan
in.Task.PlanRef = "plan-a"
in.Task.PlanMismatch = &domain.PlanMismatch{
PlanRef: "plan-a", PhaseID: "phase-3", AtSHA: "18ccaf00000000000000000000000000000000aa",
Observed: "the aggregation runs per figure, not per person",
Contradicts: "the plan states the pipeline already aggregates per person",
Evidence: []string{"internal/figures/aggregate.go:88"},
RequestedAction: domain.PlanMismatchReplan,
}
out, err := Build(in)
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
"## Why this phase reopened",
"phase: phase-3",
"the aggregation runs per figure",
"the plan states the pipeline already aggregates per person",
"internal/figures/aggregate.go:88",
"stays accepted until you seal a replacement",
} {
if !strings.Contains(out.Task, want) {
t.Fatalf("the planner was not told %q:\n%s", want, out.Task)
}
}
// It must come before the artifacts it changes the reading of.
in.Research = &workphase.Research{Findings: []workphase.Finding{{
ID: "r1", Confidence: "fact", Claim: "aggregation is per figure",
Evidence: "internal/figures/aggregate.go:88",
}}}
out, err = Build(in)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(out.Task, "## Accepted research") {
t.Fatal("the research this planner works from is missing")
}
if strings.Index(out.Task, "## Why this phase reopened") > strings.Index(out.Task, "## Accepted research") {
t.Fatal("the reason this session exists is rendered below the material it qualifies")
}
// An implementer is looking at the code, not at an account of the last
// contradiction, so it renders nowhere else.
in.Phase = domain.WorkPhaseImplement
out, err = Build(in)
if err != nil {
t.Fatal(err)
}
if strings.Contains(out.Task, "## Why this phase reopened") {
t.Fatal("a settled contradiction reached the implementation phase")
}
}
+7 -1
View File
@@ -113,6 +113,12 @@ type DebtObservation struct {
Detail string `json:"detail,omitempty"`
Paths []string `json:"paths,omitempty"`
At time.Time `json:"at"`
// Repeats is how many times this one incident recurred. It is intensity,
// never recurrence: one worker stuck in a five-second retry loop produced
// 301 repeats of a single failure, and counting those as 301 pieces of
// evidence would make one broken worker look like chronic, system-wide
// debt. Recurrence is the number of independent observations.
Repeats int `json:"repeats,omitempty"`
}
func (o DebtObservation) Validate() error {
@@ -251,7 +257,7 @@ func DebtClassForBlockReason(r BlockReason) (DebtClass, bool) {
// classes a worker actually emits are listed; an unknown one is not guessed at.
func DebtClassForFailureClass(f string) (DebtClass, bool) {
switch f {
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired":
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired", "handoff_unanswered":
return DebtOperational, true
case "invalid_handoff":
return DebtCorrectness, true
+58 -9
View File
@@ -82,6 +82,23 @@ const (
BlockReasonUnknown BlockReason = "unknown"
)
// BlockReasonAwaitsReply reports whether this stop is a question waiting on a
// human, rather than a fault or a budget. Every reason listed here resumes on
// a reply, so both the loop that goes and reads replies and the loop that
// returns answered tasks to the queue must agree on the set. They did not:
// plan_mismatch blocked for an answer that nothing ever went to fetch, and
// nothing would have resumed it if it had (F64).
//
// operator_required is deliberately absent. A task that has spent its question
// budget continues when an operator says so, not when someone replies.
func BlockReasonAwaitsReply(r BlockReason) bool {
switch r {
case BlockReasonHumanDecision, BlockReasonTrajectoryGate, BlockReasonPlanMismatch:
return true
}
return false
}
func (r BlockReason) Valid() bool {
switch r {
case BlockReasonLeaseFailure, BlockReasonWorkerOffline, BlockReasonLeaseExpired,
@@ -168,19 +185,19 @@ type Task struct {
QualityGate string `json:"quality_gate,omitempty"`
// Block evidence is projected from TaskBlocked so terminal records remain
// diagnosable after the live coordinator mapping is gone.
Blocker string `json:"blocker,omitempty"`
BlockReason BlockReason `json:"block_reason,omitempty"`
BlockedAt time.Time `json:"blocked_at,omitempty"`
LastPaneID string `json:"last_pane_id,omitempty"`
LastHarness string `json:"last_harness_id,omitempty"`
Blocker string `json:"blocker,omitempty"`
BlockReason BlockReason `json:"block_reason,omitempty"`
BlockedAt time.Time `json:"blocked_at,omitempty"`
LastPaneID string `json:"last_pane_id,omitempty"`
LastHarness string `json:"last_harness_id,omitempty"`
// LastLeaseEpoch is the fencing token of the lease that most recently
// ended. A worker can push its release anchor and only then discover the
// lease expired; the finished work is durable in git but the commit can
// never land. Retaining the epoch lets exactly that owner still commit
// while the task sits unleased.
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
LastSession SessionEvidence `json:"last_session,omitempty"`
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
LastSession SessionEvidence `json:"last_session,omitempty"`
// Recovery state is part of the durable projection, never process-local
// router memory. This makes retry and operator diagnostics survive a
// coordinator restart.
@@ -217,6 +234,13 @@ type Task struct {
// current one, oldest first. A superseded plan stays queryable: the
// verification recorded against it is provenance, not garbage.
PlanHistory []string `json:"plan_history,omitempty"`
// PlanMismatch is the contradiction that reopened this task's phase. The
// planning session convened by a replan has to be told what it is there to
// fix, and the implementer that found it is gone by then (F66). It is
// cleared when a replacement plan is accepted, or when the stop it caused
// is answered, because after either it is history rather than a live
// instruction.
PlanMismatch *PlanMismatch `json:"plan_mismatch,omitempty"`
LastError string `json:"last_error,omitempty"`
}
@@ -269,6 +293,25 @@ func NewID() string {
_, _ = rand.Read(b[6:])
return ulidEncoding.EncodeToString(b)
}
// EventWithoutTask reports whether an event records something about the
// system rather than about one task's lifecycle. These carry the "system"
// aggregate id, so no task projection has to exist for them.
//
// One list, because there were three: the replay guard, the append guard and
// the transition check each kept their own copy, and adding an event type to
// two of them left it rejected by the third.
func EventWithoutTask(typ string) bool {
switch typ {
case "QuotaReported", "StandupAdvisory", "ApprovalGranted", "ApprovalDenied",
EventObservationIncidentOpened, EventObservationIncidentClosed:
// An intervention is deliberately absent: it names a task when it
// repaired one, and that task must exist.
return true
}
return false
}
func ValidateEvent(e Event) error {
if e.SchemaVersion > CurrentEventSchema || e.Type == "" || e.TaskID == "" || len(e.Payload) == 0 || len(e.Payload) > 64*1024 {
return ErrInvalid
@@ -276,7 +319,7 @@ func ValidateEvent(e Event) error {
if e.SchemaVersion >= 2 && strings.TrimSpace(e.Surface) == "" {
return fmt.Errorf("%w: surface required", ErrInvalid)
}
allowed := map[string]bool{"TaskCreated": true, "TaskLeased": true, "TaskLeaseRenewed": true, "TaskReleased": true, "TaskLaunchAcknowledged": true, "TaskPickupValidated": true, "TaskCompleted": true, "TaskFailed": true, "TaskBlocked": true, "TaskNeedsAttention": true, "ApprovalRequested": true, "ApprovalGranted": true, "ApprovalDenied": true, "TaskAmended": true, "TaskCorrected": true, "QuotaReported": true, "StandupAdvisory": true, EventHumanDecisionRecorded: true, EventHumanDecisionSuperseded: true, EventWorkPhaseChanged: true, EventDeferredFindingRecorded: true, EventReviewRecorded: true, EventTaskSubmitted: true, EventTaskChangesRequested: true, EventPlanPhaseVerified: true, EventPlanMismatchRecorded: true}
allowed := map[string]bool{"TaskCreated": true, "TaskLeased": true, "TaskLeaseRenewed": true, "TaskReleased": true, "TaskLaunchAcknowledged": true, "TaskPickupValidated": true, "TaskCompleted": true, "TaskFailed": true, "TaskBlocked": true, "TaskNeedsAttention": true, "ApprovalRequested": true, "ApprovalGranted": true, "ApprovalDenied": true, "TaskAmended": true, "TaskCorrected": true, "QuotaReported": true, "StandupAdvisory": true, EventHumanDecisionRecorded: true, EventHumanDecisionSuperseded: true, EventWorkPhaseChanged: true, EventDeferredFindingRecorded: true, EventReviewRecorded: true, EventTaskSubmitted: true, EventTaskChangesRequested: true, EventPlanPhaseVerified: true, EventPlanMismatchRecorded: true, EventObservationIncidentOpened: true, EventObservationIncidentClosed: true, EventOperatorInterventionRecorded: true}
if !allowed[e.Type] {
return fmt.Errorf("%w: unknown type %q", ErrInvalid, e.Type)
}
@@ -558,6 +601,12 @@ func ValidatePayload(typ string, p map[string]any) error {
return ValidatePlanPhaseVerified(p)
case EventPlanMismatchRecorded:
return ValidatePlanMismatchRecorded(p)
case EventObservationIncidentOpened:
return ValidateObservationIncidentOpened(p)
case EventObservationIncidentClosed:
return ValidateObservationIncidentClosed(p)
case EventOperatorInterventionRecorded:
return ValidateOperatorInterventionRecorded(p)
case EventReviewRecorded:
if err := requiredHash(p, "artifact_ref"); err != nil {
return err
+81
View File
@@ -0,0 +1,81 @@
package domain
import (
"fmt"
"strings"
"time"
)
// EventOperatorInterventionRecorded is a human saying what they repaired by
// hand. Nothing infers it: a manual fix happens outside Orchestra by
// definition, so the only honest way to have the evidence is for the operator
// to state it. Without this the debt ledger reported manual recovery as a
// permanent gap, and every repair that kept the system running was invisible
// to the record of how much the system costs to run.
const EventOperatorInterventionRecorded = "OperatorInterventionRecorded"
// InterventionKind is what the operator did. The list is closed so the ledger
// can group repairs; an unrecognised kind is refused rather than guessed at.
type InterventionKind string
const (
InterventionWorkerRestart InterventionKind = "worker_restart"
InterventionTransactionClean InterventionKind = "transaction_cleanup"
InterventionForcedRelease InterventionKind = "forced_release"
InterventionStateRepair InterventionKind = "state_repair"
InterventionManualRequeue InterventionKind = "manual_requeue"
InterventionPhaseRecovery InterventionKind = "manual_phase_recovery"
)
func (k InterventionKind) Valid() bool {
switch k {
case InterventionWorkerRestart, InterventionTransactionClean, InterventionForcedRelease,
InterventionStateRepair, InterventionManualRequeue, InterventionPhaseRecovery:
return true
}
return false
}
// OperatorIntervention is one recorded manual repair.
type OperatorIntervention struct {
TaskID string `json:"task_id,omitempty"`
WorkerID string `json:"worker_id,omitempty"`
Kind InterventionKind `json:"kind"`
// Reason is the operator's own account of why it was needed. It is the
// part a later reader cannot reconstruct from anything else.
Reason string `json:"reason"`
// RelatedEventID and RelatedTransactionID point at what was repaired, so a
// reader can find the failure this answered rather than infer it.
RelatedEventID string `json:"related_event_id,omitempty"`
RelatedTransactionID string `json:"related_transaction_id,omitempty"`
Components []string `json:"components,omitempty"`
At time.Time `json:"at,omitempty"`
}
const maxInterventionReason = 1000
func (i OperatorIntervention) Validate() error {
if !i.Kind.Valid() {
return fmt.Errorf("%w: %q is not an intervention kind", ErrInvalid, i.Kind)
}
if strings.TrimSpace(i.Reason) == "" {
return fmt.Errorf("%w: an intervention states why it was needed", ErrInvalid)
}
if len(i.Reason) > maxInterventionReason {
return fmt.Errorf("%w: reason exceeds %d characters", ErrInvalid, maxInterventionReason)
}
if i.TaskID == "" && i.WorkerID == "" {
return fmt.Errorf("%w: an intervention names the task or the worker it repaired", ErrInvalid)
}
return nil
}
func ValidateOperatorInterventionRecorded(p map[string]any) error {
kind, _ := p["kind"].(string)
reason, _ := p["reason"].(string)
task, _ := p["task_id"].(string)
worker, _ := p["worker_id"].(string)
return OperatorIntervention{
Kind: InterventionKind(kind), Reason: reason, TaskID: task, WorkerID: worker,
}.Validate()
}
+160
View File
@@ -0,0 +1,160 @@
package domain
import (
"fmt"
"regexp"
"strings"
"time"
)
// A worker's observation ring is bounded, lossy, and local: it holds distinct
// failure messages with repeat counts and nothing else, and it disappears when
// the process does. The debt ledger reported that gap about itself, because no
// event carried any of it.
//
// These two events make it durable as incidents rather than as symptoms. Run
// 11 saw the same 409 refusal 301 times; that is one incident with an
// intensity of 301, not 301 pieces of evidence. Recurrence has to mean "this
// happened on four independent leases", or one stuck loop makes everything
// look chronic.
const (
EventObservationIncidentOpened = "ObservationIncidentOpened"
EventObservationIncidentClosed = "ObservationIncidentClosed"
)
// ObservationCloseReason is why Orchestra finalized an incident. None of them
// is "the message stopped appearing in the ring": the ring is a bounded
// history, so absence proves eviction as easily as recovery.
type ObservationCloseReason string
const (
// ObservationCloseLeaseEnd and ObservationCloseEpochChange are the natural
// boundaries of a lease-scoped incident. The work it was about is over.
ObservationCloseLeaseEnd ObservationCloseReason = "lease_end"
ObservationCloseEpochChange ObservationCloseReason = "epoch_change"
// ObservationCloseWorkerRestart ends every incident of an incarnation. A
// new process cannot continue the old one's symptom.
ObservationCloseWorkerRestart ObservationCloseReason = "worker_restart"
// ObservationCloseQuietTimeout is the only closer for an observation with
// no lease to bound it, and it fires on last_seen going stale rather than
// on the entry vanishing.
ObservationCloseQuietTimeout ObservationCloseReason = "quiet_timeout"
)
func (r ObservationCloseReason) Valid() bool {
switch r {
case ObservationCloseLeaseEnd, ObservationCloseEpochChange, ObservationCloseWorkerRestart, ObservationCloseQuietTimeout:
return true
}
return false
}
// WorkerObservation is one entry of a worker's ring as reported on a
// heartbeat. It is the input to the incident projection, never a stored event.
type WorkerObservation struct {
Message string `json:"message"`
Count int `json:"count"`
First time.Time `json:"first"`
Last time.Time `json:"last"`
}
// ObservationIncident is one durable incident: a signature seen by one worker,
// on one lease when there is one, from its first occurrence to the boundary
// that ended it.
type ObservationIncident struct {
ID string `json:"observation_id"`
WorkerID string `json:"worker_id"`
Incarnation string `json:"incarnation,omitempty"`
TaskID string `json:"task_id,omitempty"`
LeaseEpoch string `json:"lease_epoch,omitempty"`
// Signature is the message with its task ids, commit shas, paths and
// durations replaced, so the same failure on two tasks shares it. Grouping
// on the raw message would make every task its own kind of problem.
Signature string `json:"signature"`
Detail string `json:"detail,omitempty"`
FirstSeen time.Time `json:"first_seen"`
// LastSeen is the last actual occurrence. ClosedAt is when Orchestra
// finalized the incident, which is later and often much later: an incident
// stays open until its lease ends, and open means "not yet final evidence"
// rather than "happening right now".
LastSeen time.Time `json:"last_seen,omitempty"`
ClosedAt time.Time `json:"closed_at,omitempty"`
RepeatCount int `json:"repeat_count,omitempty"`
CloseReason ObservationCloseReason `json:"close_reason,omitempty"`
}
// Key identifies an incident. Two workers reporting the same failure are two
// incidents, and so are two leases of one task.
func (i ObservationIncident) Key() string {
return strings.Join([]string{i.WorkerID, i.TaskID, i.LeaseEpoch, i.Signature}, "\x00")
}
var (
// Case-insensitive: a task id appears upper-case in a message and
// lower-case inside a pane name, and the live ledger's first run showed
// pane names keeping their task, which would give the same failure a
// different signature on every task.
observationID = regexp.MustCompile(`(?i)\b[0-9A-HJKMNP-TV-Z]{26}\b`)
observationSHA = regexp.MustCompile(`\b[0-9a-f]{7,64}\b`)
observationDuration = regexp.MustCompile(`\b\d+(\.\d+)?(ns|us|µs|ms|s|m|h)(\d+(\.\d+)?(ns|us|µs|ms|s|m|h))*\b`)
observationNumber = regexp.MustCompile(`\b\d+\b`)
observationPath = regexp.MustCompile(`(/[\w.-]+){2,}`)
)
// ObservationSignature collapses one message to the kind of failure it is.
// "lease A not renewed" and "lease B not renewed" are the same problem seen
// twice, which is the whole basis of counting recurrence across tasks.
func ObservationSignature(message string) string {
s := strings.TrimSpace(message)
s = observationID.ReplaceAllString(s, "<id>")
s = observationPath.ReplaceAllString(s, "<path>")
s = observationDuration.ReplaceAllString(s, "<dur>")
s = observationSHA.ReplaceAllString(s, "<sha>")
s = observationNumber.ReplaceAllString(s, "<n>")
s = strings.Join(strings.Fields(s), " ")
if len(s) > 200 {
s = s[:200]
}
return s
}
// ObservationTaskID reads the task a failure was about out of the message
// itself. The ring is a history: it holds entries from tasks that ended long
// ago, so the worker's currently active task is the wrong answer for most of
// them, and attributing an old failure to whatever is running now would be a
// fabricated association.
func ObservationTaskID(message string) string {
if m := observationTaskID.FindString(message); m != "" {
return strings.ToUpper(m)
}
return ""
}
var observationTaskID = regexp.MustCompile(`(?i)\b[0-9A-HJKMNP-TV-Z]{26}\b`)
func ValidateObservationIncidentOpened(p map[string]any) error {
if id, _ := p["observation_id"].(string); strings.TrimSpace(id) == "" {
return fmt.Errorf("%w: observation_id required", ErrInvalid)
}
if w, _ := p["worker_id"].(string); strings.TrimSpace(w) == "" {
return fmt.Errorf("%w: worker_id required", ErrInvalid)
}
if sig, _ := p["signature"].(string); strings.TrimSpace(sig) == "" {
return fmt.Errorf("%w: signature required", ErrInvalid)
}
return nil
}
func ValidateObservationIncidentClosed(p map[string]any) error {
if id, _ := p["observation_id"].(string); strings.TrimSpace(id) == "" {
return fmt.Errorf("%w: observation_id required", ErrInvalid)
}
reason, _ := p["close_reason"].(string)
if !ObservationCloseReason(reason).Valid() {
return fmt.Errorf("%w: close_reason %q is not a close reason", ErrInvalid, reason)
}
if c, ok := p["repeat_count"].(float64); ok && c < 0 {
return fmt.Errorf("%w: repeat_count cannot be negative", ErrInvalid)
}
return nil
}
+7
View File
@@ -55,6 +55,13 @@ type PlanPhaseRecord struct {
// EvidenceRef is the CAS ref of the captured command output.
EvidenceRef string `json:"evidence_ref,omitempty"`
At time.Time `json:"at"`
// ManualAtSHA is the tree a human was actually looking at when they signed
// this phase off. A manual check on most projects is a human reading
// output, so a sign-off establishes something about one tree and nothing
// about the next one (F63). Rerunning the automated half re-establishes it
// at the new commit; the manual half has to be given again, and this is
// what makes the difference visible instead of assumed.
ManualAtSHA string `json:"manual_at_sha,omitempty"`
}
// Stale reports whether the tree has moved since this phase was verified. A
+11 -4
View File
@@ -21,10 +21,14 @@ type Worker struct {
Capacity int `json:"capacity"`
SupportedProjects []string `json:"supported_projects"`
Build buildinfo.Info `json:"build"`
LastSeen time.Time `json:"last_seen"`
Online bool `json:"online"`
Health WorkerHealth `json:"health"`
Token string `json:"-"`
// Incarnation identifies one worker process. Nothing else on the wire
// distinguishes a restarted worker from a running one, and an observation
// incident cannot outlive the process that reported it.
Incarnation string `json:"incarnation,omitempty"`
LastSeen time.Time `json:"last_seen"`
Online bool `json:"online"`
Health WorkerHealth `json:"health"`
Token string `json:"-"`
}
// WorkerHealth is reported by the worker that owns the local execution backend.
@@ -42,6 +46,9 @@ type WorkerHealth struct {
// Observations is the bounded set of distinct failures behind LastError,
// which keeps its wire name and still reports only the newest.
Observations []Observation `json:"observations,omitempty"`
// Incarnation repeats the worker's process identity on every heartbeat, so
// the coordinator sees a restart even if it missed the registration.
Incarnation string `json:"incarnation,omitempty"`
}
// Observation is one distinct worker failure with its repeat count. A single
+8
View File
@@ -174,6 +174,14 @@ type Session struct {
// its §6.1 handoff (HandoffFile) — avoids re-sending the same prompt
// every tick while Release keeps waiting for the file to appear.
HandoffRequested bool `json:"handoff_requested,omitempty"`
// HandoffRequestedAt stamps that prompt. A request nobody answers used to
// end as an ordinary idle expiry, indistinguishable from an agent that
// never started (F62); the stamp is what makes the wait bounded and the
// giving-up causal.
HandoffRequestedAt time.Time `json:"handoff_requested_at,omitempty"`
// HandoffRetried records that the request was re-sent once, so a session
// waiting on an answer is not re-prompted every tick.
HandoffRetried bool `json:"handoff_retried,omitempty"`
// HandoffReason is selected by the coordinator when it asks for the
// semantic report. The checkout worker, rather than the harness, copies
// it into the canonical handoff it seals at release time.
+10 -2
View File
@@ -43,9 +43,17 @@ func CheckDebtEligibility(item domain.DebtItem) DebtCheck {
if len(why) > 0 {
return DebtCheck{true, why}
}
// Reasons state facts about this item and nothing about the
// architecture around it. The second line here used to say manual
// interventions were recorded by no event type, which was true when it
// was written and false the moment OperatorInterventionRecorded
// landed. A decision that is right for a reason that has become a lie
// cannot be audited, and why a count is zero is not this function's
// business: no intervention happened, none was recorded, or none was
// migrated all read the same from here.
return DebtCheck{false, []string{
fmt.Sprintf("needs 3 occurrences across 2 tasks, or 1 manual intervention; has %d across %d tasks with %d interventions", recurrence, tasks, manual),
"manual interventions are not recorded by any event type, so that count reads 0 on every current log",
fmt.Sprintf("breadth threshold not met: %d occurrences across %d tasks, needs 3 across 2", recurrence, tasks),
fmt.Sprintf("manual intervention threshold not met: %d recorded, needs 1", manual),
}}
case domain.DebtStructural:
if recurrence >= 3 {
+12 -2
View File
@@ -62,7 +62,17 @@ func TestDebtRefusalNamesTheMissingEvidence(t *testing.T) {
t.Fatalf("want a refusal with reasons, got %+v", check)
}
joined := strings.Join(check.Reasons, " ")
if !strings.Contains(joined, "manual interventions are not recorded") {
t.Fatalf("the refusal must say the intervention count is structurally zero: %v", check.Reasons)
// Every reason is a fact about this item. The old text asserted that no
// event type recorded manual interventions, which stopped being true the
// day one did, leaving a correct decision explained by a lie.
for _, want := range []string{"1 occurrences across 1 tasks", "0 recorded, needs 1"} {
if !strings.Contains(joined, want) {
t.Fatalf("the refusal does not state %q: %v", want, check.Reasons)
}
}
for _, forbidden := range []string{"event type", "every current log"} {
if strings.Contains(joined, forbidden) {
t.Fatalf("a reason describes the architecture instead of the item: %v", check.Reasons)
}
}
}
+19 -12
View File
@@ -58,18 +58,27 @@ func RequestHumanDecision(s *store.Store, project registry.Project, taskID strin
return blockTask(s, t, domain.BlockReasonHumanDecision, req.Render(), &req)
}
// fenceToLease binds a lifecycle event to the lease that is producing it.
// Store.Append fences every lifecycle event on a leased task against the
// current owner and epoch, so an event that omits them is a conflict rather
// than a block. Every coordinator-side stop goes through here: two of them did
// not, and both failed silently against a live lease (F65). A task stops for a
// human only while some session is running, so the leased case is the only one
// that ever mattered.
func fenceToLease(payload map[string]any, t domain.Task) {
if t.Lease == nil {
return
}
payload["harness_id"] = t.Lease.HarnessID
payload["lease_epoch"] = t.Lease.Epoch
}
func blockTask(s *store.Store, t domain.Task, reason domain.BlockReason, blocker string, req *domain.DecisionRequest) (domain.Event, error) {
payload := map[string]any{
"blocker": blocker, "block_reason": string(reason),
"lifecycle_phase": "awaiting_human",
}
if t.Lease != nil {
// Store.Append fences every lifecycle event on a leased task against
// the current owner and epoch. A question from a session that no
// longer owns the task is a conflict, not a block.
payload["harness_id"] = t.Lease.HarnessID
payload["lease_epoch"] = t.Lease.Epoch
}
fenceToLease(payload, t)
if req != nil {
payload["decision_request"] = req
}
@@ -107,11 +116,9 @@ func ResumeAnsweredBlockers(s *store.Store) ([]domain.Event, error) {
if t.State != domain.StateBlocked {
continue
}
switch t.BlockReason {
case domain.BlockReasonHumanDecision, domain.BlockReasonTrajectoryGate:
default:
// operator_required is deliberately not resumed by a reply. An
// operator decides when a task that spent its budget continues.
// operator_required is deliberately not resumed by a reply. An
// operator decides when a task that spent its budget continues.
if !domain.BlockReasonAwaitsReply(t.BlockReason) {
continue
}
if !blockerAnswered(s, t.ID, t.BlockReason) {
+275
View File
@@ -0,0 +1,275 @@
package operations
import (
"encoding/json"
"fmt"
"time"
"orchestra/internal/authz"
"orchestra/internal/domain"
"orchestra/internal/store"
)
// QuietTimeout bounds an incident that has no lease to bound it. A worker-level
// observation has no terminal boundary of its own, so staleness of its last
// actual occurrence is the only honest closer.
const QuietTimeout = 10 * time.Minute
// ObservationTracker turns a worker's bounded, lossy ring into durable
// incidents. It is deliberately not a copy of the ring.
//
// The rules that matter, and why:
//
// - An incident is opened at first sight and appended immediately, so a
// coordinator that dies mid-incident still leaves the fact that it existed.
// - A repeat updates the aggregate and appends nothing. Run 11's 409 loop
// repeated 301 times; appending each would have been 301 pieces of evidence
// for one problem, and would have made every debt item eligible at once.
// - Absence from the ring closes nothing. The ring is a bounded history, so a
// message can vanish because it was evicted rather than because it stopped.
// - The lease is the scope. The same signature going quiet and returning
// inside one epoch is one incident, not two recurrences.
type ObservationTracker struct {
Store *store.Store
// counts is the last count this tracker saw for an open incident, so a
// ring entry that is evicted and recreated accumulates rather than
// restarting. Reported 34, evicted, reported 3 again means 37 occurrences,
// not 3. In-memory: a restart loses the accumulation, never the incident.
counts map[string]int
// seen is the high-water mark per worker and signature, kept after an
// incident closes. The ring is a history, so a closed incident's entry
// keeps being reported for as long as it survives eviction; without this,
// every heartbeat after a quiet timeout opened the same incident again and
// manufactured recurrence out of one old failure. Live on the first real
// run: three signatures, four incidents each, none of them a new event.
seen map[string]watermark
// incarnations is the last incarnation seen per worker, which is what makes
// a restart detectable at all.
incarnations map[string]string
}
// watermark is the last occurrence this tracker accounted for one worker's
// signature, whether or not its incident is still open.
type watermark struct {
last time.Time
count int
}
// WorkerReport is one heartbeat's worth of attributed observations. The
// coordinator attributes them, because the ring carries only messages: the
// worker's active task and that task's current lease epoch are what bind an
// incident to the work it happened during.
type WorkerReport struct {
WorkerID string
Incarnation string
TaskID string
LeaseEpoch string
Observations []domain.WorkerObservation
At time.Time
}
// Ingest folds one heartbeat into the durable incidents and returns the events
// it appended. Every close it decides is one of the four boundaries; none of
// them is "the message is no longer in the ring".
func (t *ObservationTracker) Ingest(r WorkerReport) ([]domain.Event, error) {
if t == nil || t.Store == nil || r.WorkerID == "" {
return nil, nil
}
if t.counts == nil {
t.counts, t.incarnations = map[string]int{}, map[string]string{}
t.seen = map[string]watermark{}
}
at := r.At
if at.IsZero() {
at = time.Now().UTC()
}
var appended []domain.Event
// A new process cannot continue the previous one's symptom, so its
// incidents are finalized before anything this heartbeat says is folded in.
if previous, seen := t.incarnations[r.WorkerID]; r.Incarnation != "" && seen && previous != r.Incarnation {
closed, err := t.closeWhere(at, domain.ObservationCloseWorkerRestart, func(inc domain.ObservationIncident) bool {
return inc.WorkerID == r.WorkerID
})
appended = append(appended, closed...)
if err != nil {
return appended, err
}
}
if r.Incarnation != "" {
t.incarnations[r.WorkerID] = r.Incarnation
}
open := map[string]domain.ObservationIncident{}
for _, inc := range t.Store.OpenObservations() {
if inc.WorkerID == r.WorkerID {
open[inc.Key()] = inc
}
}
for _, o := range r.Observations {
signature := domain.ObservationSignature(o.Message)
if signature == "" {
continue
}
// The task comes from the message, because the ring outlives the work
// it describes. Only a failure that names no task is attributed to the
// lease running now.
taskID, epoch := domain.ObservationTaskID(o.Message), ""
if taskID == "" {
taskID, epoch = r.TaskID, r.LeaseEpoch
} else if taskID == r.TaskID {
epoch = r.LeaseEpoch
}
candidate := domain.ObservationIncident{
WorkerID: r.WorkerID, Incarnation: r.Incarnation, Signature: signature,
TaskID: taskID, LeaseEpoch: epoch,
}
mark := t.seen[r.WorkerID+"\x00"+signature]
existing, isOpen := open[candidate.Key()]
if !isOpen {
// Nothing new: this is a closed incident's entry still sitting in
// the ring. Presence is not occurrence.
if !firstOr(o.Last, at).After(mark.last) && o.Count <= mark.count {
continue
}
candidate.ID = domain.NewID()
candidate.Detail = o.Message
candidate.FirstSeen = firstOr(o.First, at)
candidate.LastSeen = firstOr(o.Last, at)
e, err := t.append(domain.EventObservationIncidentOpened, candidate)
if err != nil {
return appended, err
}
appended = append(appended, e)
t.counts[candidate.ID] = o.Count
t.seen[r.WorkerID+"\x00"+signature] = watermark{last: candidate.LastSeen, count: o.Count}
continue
}
// Open already: accumulate, append nothing. A count lower than the last
// one means the ring evicted the entry and started it again.
delta := o.Count - t.counts[existing.ID]
if delta < 0 {
delta = o.Count
}
t.counts[existing.ID] += delta
if last := firstOr(o.Last, at); last.After(existing.LastSeen) {
existing.LastSeen = last
t.Store.NoteObservation(existing) // last_seen is durable at close
}
t.seen[r.WorkerID+"\x00"+signature] = watermark{last: existing.LastSeen, count: o.Count}
}
// The boundaries. A lease that ended, an epoch that changed, and a
// worker-level incident whose last occurrence has gone stale.
closed, err := t.closeWhere(at, "", func(inc domain.ObservationIncident) bool {
if inc.WorkerID != r.WorkerID {
return false
}
// Only an incident bound to a live lease has a lease boundary to close
// it. One read out of the ring about a task that already finished has
// no such boundary, so it ends the way a worker-level incident does.
if inc.LeaseEpoch == "" {
return at.Sub(inc.LastSeen) > QuietTimeout
}
return inc.TaskID != r.TaskID || inc.LeaseEpoch != r.LeaseEpoch
})
appended = append(appended, closed...)
return appended, err
}
// closeWhere finalizes every open incident the predicate selects. A reason of
// "" is resolved per incident, which is what lets one sweep close a lease that
// ended and a worker-level incident that went quiet.
func (t *ObservationTracker) closeWhere(at time.Time, reason domain.ObservationCloseReason, match func(domain.ObservationIncident) bool) ([]domain.Event, error) {
var out []domain.Event
for _, inc := range t.Store.OpenObservations() {
if !match(inc) {
continue
}
inc.ClosedAt = at
inc.RepeatCount = t.counts[inc.ID]
inc.CloseReason = reason
if inc.CloseReason == "" {
switch {
case inc.LeaseEpoch == "":
inc.CloseReason = domain.ObservationCloseQuietTimeout
case inc.TaskID != "":
inc.CloseReason = domain.ObservationCloseEpochChange
default:
inc.CloseReason = domain.ObservationCloseLeaseEnd
}
}
e, err := t.append(domain.EventObservationIncidentClosed, inc)
if err != nil {
return out, err
}
delete(t.counts, inc.ID)
out = append(out, e)
}
return out, nil
}
// append writes the incident as a worker-scoped event. The task it happened
// during is carried in the payload rather than in Event.TaskID on purpose: an
// incident is evidence about a worker, and binding it to the task aggregate
// would bump that task's version from a path the lease knows nothing about.
func (t *ObservationTracker) append(typ string, inc domain.ObservationIncident) (domain.Event, error) {
b, err := json.Marshal(inc)
if err != nil {
return domain.Event{}, err
}
// "system" is the same aggregate QuotaReported uses for worker-scoped
// facts: every event needs a task id, and this evidence belongs to a
// worker rather than to any one task.
e := domain.Event{ID: domain.NewID(), Type: typ, TaskID: "system", Payload: b, Surface: string(authz.System)}
if err := t.Store.Append(e); err != nil {
return domain.Event{}, fmt.Errorf("record observation incident: %w", err)
}
return e, nil
}
func firstOr(t, fallback time.Time) time.Time {
if t.IsZero() {
return fallback
}
return t
}
// RecordIntervention writes down a repair the operator made by hand. It is
// deliberately an explicit act: Orchestra cannot see a worker someone
// restarted or a transaction someone deleted, and inferring "an operator
// probably intervened" from a gap in the log would put guesses into the
// evidence the ledger is built from.
func RecordIntervention(s *store.Store, surface authz.Surface, in domain.OperatorIntervention) (domain.Event, error) {
if err := in.Validate(); err != nil {
return domain.Event{}, err
}
if in.At.IsZero() {
in.At = time.Now().UTC()
}
taskID := in.TaskID
version := 0
if taskID != "" {
t, ok := s.Task(taskID)
if !ok {
return domain.Event{}, domain.ErrNotFound
}
version = t.Version + 1
} else {
// A repair with no task is still about this deployment, so it lands on
// the same aggregate the other worker-scoped facts use.
taskID = "system"
}
b, err := json.Marshal(in)
if err != nil {
return domain.Event{}, err
}
e := domain.Event{
ID: domain.NewID(), Type: domain.EventOperatorInterventionRecorded,
TaskID: taskID, Version: version, At: in.At, Payload: b, Surface: string(surface),
}
if err := s.Append(e); err != nil {
return domain.Event{}, err
}
return e, nil
}
+295
View File
@@ -0,0 +1,295 @@
package operations
import (
"encoding/json"
"testing"
"time"
"orchestra/internal/domain"
"orchestra/internal/store"
)
func tracker(t *testing.T) (*ObservationTracker, *store.Store) {
t.Helper()
s, err := store.Open(t.TempDir())
if err != nil {
t.Fatal(err)
}
return &ObservationTracker{Store: s}, s
}
func ring(message string, count int, last time.Time) []domain.WorkerObservation {
return []domain.WorkerObservation{{Message: message, Count: count, First: last.Add(-time.Minute), Last: last}}
}
func closedIncident(t *testing.T, s *store.Store) domain.ObservationIncident {
t.Helper()
var out domain.ObservationIncident
found := 0
for _, e := range s.Events(0) {
if e.Type != domain.EventObservationIncidentClosed {
continue
}
found++
if err := json.Unmarshal(e.Payload, &out); err != nil {
t.Fatal(err)
}
}
if found != 1 {
t.Fatalf("closed incidents = %d, want 1", found)
}
return out
}
// 301 repeats of one refusal are one incident with an intensity of 301, not
// 301 pieces of evidence. Appending each would spam the log and make one stuck
// loop look like chronic, recurring debt.
func TestRepeatsAreOneIncident(t *testing.T) {
tr, s := tracker(t)
at := time.Unix(1700000000, 0).UTC()
report := func(count int, when time.Time) {
if _, err := tr.Ingest(WorkerReport{
WorkerID: "workpc-claude", Incarnation: "boot-1", TaskID: "task-a", LeaseEpoch: "epoch-1",
Observations: ring("release task-a commit: 409 superseded", count, when), At: when,
}); err != nil {
t.Fatal(err)
}
}
report(1, at)
report(40, at.Add(time.Minute))
report(301, at.Add(2*time.Minute))
opened := 0
for _, e := range s.Events(0) {
if e.Type == domain.EventObservationIncidentOpened {
opened++
}
if e.Type == domain.EventObservationIncidentClosed {
t.Fatal("an incident was closed while its lease was still running")
}
}
if opened != 1 {
t.Fatalf("opened %d incidents for one repeating failure", opened)
}
if open := s.OpenObservations(); len(open) != 1 || open[0].TaskID != "task-a" {
t.Fatalf("open incidents = %+v", open)
}
}
// The ring is a bounded history, so an entry that disappears may have been
// evicted rather than resolved. Absence must not close anything, and a
// recreated entry must accumulate rather than restart its count.
func TestEvictionNeitherClosesNorRestartsTheCount(t *testing.T) {
tr, s := tracker(t)
at := time.Unix(1700000000, 0).UTC()
send := func(obs []domain.WorkerObservation, when time.Time) {
if _, err := tr.Ingest(WorkerReport{
WorkerID: "w", Incarnation: "boot-1", TaskID: "task-a", LeaseEpoch: "epoch-1",
Observations: obs, At: when,
}); err != nil {
t.Fatal(err)
}
}
send(ring("lease task-a not renewed: agent idle", 34, at), at)
// Evicted: the message is simply gone from this heartbeat.
send(nil, at.Add(time.Minute))
if len(s.OpenObservations()) != 1 {
t.Fatal("an incident was closed because its message left a bounded ring")
}
// Recreated, counting from scratch on the worker side.
send(ring("lease task-a not renewed: agent idle", 3, at.Add(2*time.Minute)), at.Add(2*time.Minute))
// The lease ends, which is a real boundary.
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "boot-1", At: at.Add(3 * time.Minute)}); err != nil {
t.Fatal(err)
}
inc := closedIncident(t, s)
if inc.RepeatCount != 37 {
t.Fatalf("repeat_count = %d, want 37 (34 before eviction plus 3 after)", inc.RepeatCount)
}
if inc.CloseReason != domain.ObservationCloseEpochChange {
t.Fatalf("close_reason = %q", inc.CloseReason)
}
if !inc.LastSeen.Equal(at.Add(2 * time.Minute)) {
t.Fatalf("last_seen = %s, want the last actual occurrence", inc.LastSeen)
}
if !inc.ClosedAt.After(inc.LastSeen) {
t.Fatal("closed_at must be when Orchestra finalized it, not when the failure last happened")
}
}
// Recurrence is independent incidents. The same signature on two tasks is two,
// which is the evidence that means something; repeats inside one are intensity.
func TestTheSameSignatureOnAnotherTaskIsASecondIncident(t *testing.T) {
tr, s := tracker(t)
at := time.Unix(1700000000, 0).UTC()
if _, err := tr.Ingest(WorkerReport{
WorkerID: "w", Incarnation: "boot-1", TaskID: "task-a", LeaseEpoch: "e1",
Observations: ring("lease task-a not renewed: agent idle", 5, at), At: at,
}); err != nil {
t.Fatal(err)
}
if _, err := tr.Ingest(WorkerReport{
WorkerID: "w", Incarnation: "boot-1", TaskID: "task-b", LeaseEpoch: "e2",
Observations: ring("lease task-b not renewed: agent idle", 2, at.Add(time.Minute)), At: at.Add(time.Minute),
}); err != nil {
t.Fatal(err)
}
opened, closed := 0, 0
for _, e := range s.Events(0) {
switch e.Type {
case domain.EventObservationIncidentOpened:
opened++
case domain.EventObservationIncidentClosed:
closed++
}
}
if opened != 2 {
t.Fatalf("opened = %d, want one incident per lease", opened)
}
if closed != 1 {
t.Fatalf("closed = %d, want the first lease finalized when the second began", closed)
}
}
// A restart cannot continue the previous process's symptom.
func TestAWorkerRestartClosesItsIncidents(t *testing.T) {
tr, s := tracker(t)
at := time.Unix(1700000000, 0).UTC()
if _, err := tr.Ingest(WorkerReport{
WorkerID: "w", Incarnation: "boot-1", TaskID: "task-a", LeaseEpoch: "e1",
Observations: ring("herdr unreachable", 9, at), At: at,
}); err != nil {
t.Fatal(err)
}
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "boot-2", At: at.Add(time.Minute)}); err != nil {
t.Fatal(err)
}
inc := closedIncident(t, s)
if inc.CloseReason != domain.ObservationCloseWorkerRestart || inc.RepeatCount != 9 {
t.Fatalf("incident = %+v", inc)
}
}
// An observation with no lease has no terminal boundary, so staleness of its
// last actual occurrence is what ends it.
func TestAWorkerLevelIncidentClosesOnQuietTimeout(t *testing.T) {
tr, s := tracker(t)
at := time.Unix(1700000000, 0).UTC()
if _, err := tr.Ingest(WorkerReport{
WorkerID: "w", Incarnation: "boot-1",
Observations: ring("heartbeat: connection refused", 4, at), At: at,
}); err != nil {
t.Fatal(err)
}
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "boot-1", At: at.Add(time.Minute)}); err != nil {
t.Fatal(err)
}
if len(s.OpenObservations()) != 1 {
t.Fatal("a worker-level incident closed before its quiet timeout")
}
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "boot-1", At: at.Add(QuietTimeout + time.Minute)}); err != nil {
t.Fatal(err)
}
if inc := closedIncident(t, s); inc.CloseReason != domain.ObservationCloseQuietTimeout {
t.Fatalf("close_reason = %q", inc.CloseReason)
}
}
// The signature is what makes recurrence countable across tasks.
func TestSignatureCollapsesIdsAndCounts(t *testing.T) {
a := domain.ObservationSignature("lease 06G4WJ9T4F35NZC4Z8QQXM9Z6G not renewed: agent status idle and pane unchanged")
b := domain.ObservationSignature("lease 06G4VF5HZW7Q4JBM3TTY7W1Y64 not renewed: agent status idle and pane unchanged")
if a != b {
t.Fatalf("the same failure on two tasks has two signatures:\n%s\n%s", a, b)
}
if c := domain.ObservationSignature("release 06G4WJ9T4F35NZC4Z8QQXM9Z6G commit: 409 superseded"); c == a {
t.Fatal("two different failures collapsed to one signature")
}
}
// The first live run of the ledger caught this: a pane name carries the task
// id in lower case, so the same failure signed differently on every task and
// could never accumulate recurrence.
func TestSignatureCollapsesAPaneName(t *testing.T) {
a := domain.ObservationSignature(`phase rotation 06G4XAFH1MBPC35VSJN7V3NS14: pane orchestra-06g4xafh1mbpc35vsjn7v3ns14-be13b045:1.0 still holds input`)
b := domain.ObservationSignature(`phase rotation 06G4WW6TND26M16CZA6WE5T458: pane orchestra-06g4ww6tnd26m16cza6we5t458-4d839c05:1.0 still holds input`)
if a != b {
t.Fatalf("one failure has two signatures:\n%s\n%s", a, b)
}
}
// The first live run manufactured recurrence out of one old failure: a quiet
// timeout closed the incident, the entry stayed in the ring because the ring
// is a bounded history rather than a set of live conditions, and every later
// heartbeat opened it again. Four incidents, one failure, no new occurrence.
func TestAClosedIncidentDoesNotReopenFromAStaleRingEntry(t *testing.T) {
tr, s := tracker(t)
at := time.Unix(1700000000, 0).UTC()
entry := ring("heartbeat: connection refused", 4, at)
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "b1", Observations: entry, At: at}); err != nil {
t.Fatal(err)
}
// Long enough to close on quiet timeout, with the entry still reported.
quiet := at.Add(QuietTimeout + time.Minute)
for i := 0; i < 4; i++ {
if _, err := tr.Ingest(WorkerReport{
WorkerID: "w", Incarnation: "b1", Observations: entry,
At: quiet.Add(time.Duration(i) * time.Minute),
}); err != nil {
t.Fatal(err)
}
}
opened, closed := 0, 0
for _, e := range s.Events(0) {
switch e.Type {
case domain.EventObservationIncidentOpened:
opened++
case domain.EventObservationIncidentClosed:
closed++
}
}
if opened != 1 || closed != 1 {
t.Fatalf("opened=%d closed=%d for one failure that never happened again", opened, closed)
}
// A real new occurrence, which the ring shows by advancing the entry.
later := quiet.Add(time.Hour)
if _, err := tr.Ingest(WorkerReport{
WorkerID: "w", Incarnation: "b1", Observations: ring("heartbeat: connection refused", 5, later), At: later,
}); err != nil {
t.Fatal(err)
}
reopened := 0
for _, e := range s.Events(0) {
if e.Type == domain.EventObservationIncidentOpened {
reopened++
}
}
if reopened != 2 {
t.Fatalf("a genuine new occurrence did not open an incident: opened=%d", reopened)
}
}
// The ring outlives the work it describes, so the task comes from the message
// rather than from whatever the worker happens to be running now.
func TestTheTaskComesFromTheMessageNotTheCurrentLease(t *testing.T) {
tr, s := tracker(t)
at := time.Unix(1700000000, 0).UTC()
if _, err := tr.Ingest(WorkerReport{
WorkerID: "w", Incarnation: "b1", TaskID: "06G4XAFH1MBPC35VSJN7V3NS14", LeaseEpoch: "now",
Observations: ring("renew lease 06G4WW6TND26M16CZA6WE5T458: 409 conflict", 3, at), At: at,
}); err != nil {
t.Fatal(err)
}
open := s.OpenObservations()
if len(open) != 1 {
t.Fatalf("open = %+v", open)
}
if open[0].TaskID != "06G4WW6TND26M16CZA6WE5T458" {
t.Fatalf("the failure was attributed to the wrong task: %q", open[0].TaskID)
}
if open[0].LeaseEpoch != "" {
t.Fatalf("an old failure inherited the current lease's epoch: %q", open[0].LeaseEpoch)
}
}
+4 -2
View File
@@ -154,11 +154,13 @@ func blockForPlanMismatch(s *store.Store, taskID string, m domain.PlanMismatch)
packet += "- evidence: " + oneLine(e) + "\n"
}
packet += "\nReply to say how to proceed. Your reply becomes a recorded decision and outranks the plan. If it resolves the contradiction, the task resumes on the same plan; say so explicitly if you want the plan rewritten instead.\n"
b, err := json.Marshal(map[string]any{
payload := map[string]any{
"blocker": packet,
"block_reason": string(domain.BlockReasonPlanMismatch),
"lifecycle_phase": "awaiting_human",
})
}
fenceToLease(payload, t)
b, err := json.Marshal(payload)
if err != nil {
return err
}
+28
View File
@@ -108,6 +108,11 @@ func TestReplanKeepsTheOldPlanUntilAReplacementIsSealed(t *testing.T) {
if len(during.PlanHistory) != 0 {
t.Fatalf("the plan was moved to history early: %v", during.PlanHistory)
}
// F66: the planning session this reopen convenes has to be told what it is
// there to fix, and the implementer that found it is gone by then.
if during.PlanMismatch == nil || during.PlanMismatch.PhaseID != "phase-1" {
t.Fatalf("the contradiction did not survive to the phase it reopened: %+v", during.PlanMismatch)
}
}
// Sealing the replacement is the moment the old plan is superseded. Progress
@@ -151,6 +156,11 @@ func TestSealingTheReplacementSupersedesThePlanAndItsProgress(t *testing.T) {
if _, err := s.Artifact(oldRef); err != nil {
t.Fatalf("the superseded plan is unreadable: %v", err)
}
// The replacement answers the contradiction, so it stops being a live
// instruction and stays in the log as history.
if after, _ := s.Task(id); after.PlanMismatch != nil {
t.Fatalf("a settled contradiction is still live: %+v", after.PlanMismatch)
}
assertPhase(t, s, id, domain.WorkPhaseImplement)
}
@@ -215,7 +225,25 @@ func TestHumanAnswerResumesTheSamePlanWithoutResealing(t *testing.T) {
if !PlanMismatchAnswered(s, id) {
t.Fatal("the human answered and the task is still waiting")
}
// F64: asserting the predicate is not asserting the resume. This test
// passed for as long as the predicate had no caller, while a task blocked
// on a mismatch stayed blocked forever however the human replied.
if events, err := ResumeAnsweredBlockers(s); err != nil || len(events) != 1 {
t.Fatalf("an answered mismatch did not return to the queue: events=%v err=%v", events, err)
}
after, _ := s.Task(id)
if after.State != domain.StateQueued {
t.Fatalf("state = %s after the human answered, want queued", after.State)
}
// F67: the stop is over, so the projection must not keep reporting it.
if after.BlockReason != "" || after.Blocker != "" {
t.Fatalf("a resumed task still reports its blocker: %q %q", after.BlockReason, after.Blocker)
}
// The answer outranks the plan and stands as an ordinary decision, so the
// contradiction it settled is not carried into later sessions either.
if after.PlanMismatch != nil {
t.Fatalf("an answered contradiction is still live: %+v", after.PlanMismatch)
}
if after.PlanRef != planRef {
t.Fatal("answering the question replaced the plan")
}
+28 -5
View File
@@ -43,6 +43,13 @@ func PlanPhaseCommands(s *store.Store, project registry.Project, taskID, phaseID
if t.PlanRef == "" {
return workphase.PlanPhase{}, fmt.Errorf("%w: this task has no accepted plan", ErrPlanPhase)
}
// Verification is implementation work. A request that arrives in another
// phase belongs to a trajectory Orchestra has already ended: run 20's
// reopened planning session executed the implementer's leftover request
// and recorded a phase of a plan that was being replaced.
if current(t) != domain.WorkPhaseImplement {
return workphase.PlanPhase{}, fmt.Errorf("%w: phase verification belongs to the implement phase, and this task is in %s", ErrPlanPhase, current(t))
}
raw, err := s.Artifact(t.PlanRef)
if err != nil {
return workphase.PlanPhase{}, fmt.Errorf("read accepted plan: %w", err)
@@ -124,7 +131,12 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
// was established.
record.Status = domain.PlanPhaseAwaitingManual
}
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID) {
// Carry the confirmed tree forward as provenance. Without it a second
// rerun would compare against nothing and re-inherit the sign-off.
if prior, ok := t.PlanPhase(phaseID); ok {
record.ManualAtSHA = prior.ManualAtSHA
}
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID, record) {
record.Status = domain.PlanPhaseVerified
}
if ref, err := s.PutArtifact(verificationEvidence(runs)); err == nil {
@@ -135,7 +147,7 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
payload := map[string]any{
"plan_ref": record.PlanRef, "phase_id": record.PhaseID, "status": string(record.Status),
"commands": record.Commands, "exit_codes": record.ExitCodes, "at_sha": record.AtSHA,
"evidence_ref": record.EvidenceRef, "at": record.At,
"evidence_ref": record.EvidenceRef, "at": record.At, "manual_at_sha": record.ManualAtSHA,
}
if t.Lease != nil {
payload["harness_id"], payload["lease_epoch"] = t.Lease.HarnessID, t.Lease.Epoch
@@ -149,9 +161,20 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
}
// manuallySignedOff reports whether a human has already approved this exact
// phase of this exact plan. The subject carries both, so a later "looks good"
// on an unrelated thread cannot satisfy a gate nobody was discussing.
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string) bool {
// phase of this exact plan, against the tree this run examined. The subject
// carries plan and phase, so a later "looks good" on an unrelated thread
// cannot satisfy a gate nobody was discussing.
//
// The tree matters as much as the subject (F63). A sign-off is a human saying
// they read what this code prints; an edit afterwards can change exactly that.
// A record whose ManualAtSHA names a different commit is therefore not signed
// off, and waits for the human again. A sign-off given before any run has no
// confirmed tree to compare against and still counts, which keeps the ordinary
// ordering unchanged.
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string, record domain.PlanPhaseRecord) bool {
if record.ManualAtSHA != "" && record.ManualAtSHA != record.AtSHA {
return false
}
intent, err := s.EffectiveIntent(t.ID)
if err != nil {
return false
+93 -1
View File
@@ -13,6 +13,7 @@ import (
const shaOne = "1111111111111111111111111111111111111111"
const shaTwo = "2222222222222222222222222222222222222222"
const shaThree = "3333333333333333333333333333333333333333"
func planProject() registry.Project {
p := registry.Project{
@@ -27,6 +28,11 @@ func planProject() registry.Project {
func planWith(t *testing.T, markdown string) (*store.Store, registry.Project, string) {
t.Helper()
s, id := phaseStore(t)
// Leased, because everything these tests drive comes from a live implement
// session. Skipping it hid F65: two coordinator-side stops omitted the
// fencing fields Store.Append requires on a leased task, and every test
// passed because no test ever leased one.
lease(t, s, id)
project := planProject()
if _, err := AdvanceWorkPhase(s, project, id, nil); err != nil {
t.Fatal(err)
@@ -251,6 +257,14 @@ func TestLegacyPlanIsExplicitlyNonProgressable(t *testing.T) {
// signOff records a human decision bound to one phase of one plan, which is
// the only thing that satisfies a manual verification gate.
func signOff(t *testing.T, s *store.Store, taskID, subject string) {
t.Helper()
signOffFrom(t, s, taskID, subject, "signoff-"+subject)
}
// signOffFrom names the comment the sign-off came from. Two sign-offs on one
// subject are a real sequence once a rerun sends a phase back to the human,
// and provenance is unique per comment.
func signOffFrom(t *testing.T, s *store.Store, taskID, subject, externalID string) {
t.Helper()
task, _ := s.Task(taskID)
if err := s.Append(domain.Event{
@@ -259,9 +273,87 @@ func signOff(t *testing.T, s *store.Store, taskID, subject string) {
Payload: mustJSONBytes(t, map[string]any{
"decision_id": domain.NewID(), "kind": "answer", "subject": subject,
"value": "manual steps confirmed",
"source": map[string]any{"provider": "gitea", "external_id": "signoff-" + subject},
"source": map[string]any{"provider": "gitea", "external_id": externalID},
}),
}); err != nil {
t.Fatalf("sign off: %v", err)
}
}
// F63, found live on run 19. A manual sign-off says a human read what this
// code prints. An edit afterwards can change exactly that, so rerunning the
// automated half at a new commit must not carry the human half with it.
func TestASignOffDoesNotSurviveTheTreeItWasGivenAgainst(t *testing.T) {
s, project, id := planWith(t, twoPhasePlan)
run := []VerificationRun{{Command: []string{"go", "test", "./internal/..."}, ExitCode: 0}}
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaOne, run); err != nil {
t.Fatal(err)
}
task, _ := s.Task(id)
signOff(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"))
task, _ = s.Task(id)
rec, _ := task.PlanPhase("phase-2")
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaOne {
t.Fatalf("sign-off did not bind to the tree it read: %+v", rec)
}
// The tree moves and the phase is re-verified. The commands pass again;
// the human has not seen the new output.
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaTwo, run); err != nil {
t.Fatal(err)
}
task, _ = s.Task(id)
rec, _ = task.PlanPhase("phase-2")
if rec.Status != domain.PlanPhaseAwaitingManual {
t.Fatalf("status = %q at a tree the human never saw, want awaiting_manual_verification", rec.Status)
}
if rec.ManualAtSHA != shaOne {
t.Fatalf("the confirmed tree was lost: %+v", rec)
}
// A second rerun must not re-inherit it either, which is what carrying
// ManualAtSHA forward is for.
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaThree, run); err != nil {
t.Fatal(err)
}
task, _ = s.Task(id)
rec, _ = task.PlanPhase("phase-2")
if rec.Status != domain.PlanPhaseAwaitingManual {
t.Fatalf("a second rerun re-inherited the sign-off: %q", rec.Status)
}
// Signing off again, on the tree that is now current, verifies it.
signOffFrom(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"), "signoff-second")
task, _ = s.Task(id)
rec, _ = task.PlanPhase("phase-2")
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaThree {
t.Fatalf("a fresh sign-off did not verify the current tree: %+v", rec)
}
}
// Run 20: a replan reopened the plan phase, the implementer's leftover
// verification request outlived its session, and the planning session that
// replaced it executed the request. Orchestra recorded a verified phase of the
// plan it was in the middle of replacing.
func TestVerificationIsRefusedOutsideImplement(t *testing.T) {
s, project, id := planWith(t, twoPhasePlan)
task, _ := s.Task(id)
m := mismatch(task.PlanRef)
m.RequestedAction = domain.PlanMismatchReplan
if _, err := RecordPlanMismatch(s, project, id, m, shaOne); err != nil {
t.Fatal(err)
}
assertPhase(t, s, id, domain.WorkPhasePlan)
_, err := RecordPlanPhaseVerification(s, project, id, "phase-1", shaOne,
[]VerificationRun{{Command: []string{"go", "build", "./..."}, ExitCode: 0}})
if !errors.Is(err, ErrPlanPhase) {
t.Fatalf("a reopened task verified a phase of the plan being replaced: %v", err)
}
if !strings.Contains(err.Error(), "implement") {
t.Fatalf("the refusal does not say which phase owns verification: %v", err)
}
if after, _ := s.Task(id); len(after.PlanPhases()) != 0 {
t.Fatalf("progress was recorded anyway: %+v", after.PlanPhases())
}
}
+4 -2
View File
@@ -67,11 +67,13 @@ func raiseTrajectoryGate(s *store.Store, t domain.Task, from, to domain.WorkPhas
if err != nil {
return err
}
b, err := json.Marshal(map[string]any{
payload := map[string]any{
"blocker": packet,
"block_reason": string(domain.BlockReasonTrajectoryGate),
"lifecycle_phase": "awaiting_human",
})
}
fenceToLease(payload, t)
b, err := json.Marshal(payload)
if err != nil {
return err
}
+36 -7
View File
@@ -102,6 +102,36 @@ func projectDebt(events []domain.Event, readArtifact func(string) ([]byte, error
o.Detail = str(p["last_error"])
add(class, domain.DebtSignature(class, failure, harness, "lease"), o,
"tasks end in "+failure, review.Important)
case domain.EventObservationIncidentClosed:
// One incident, whatever it repeated. The debt class comes from
// the signature's shape rather than a failure class, because a
// worker observation is a symptom the worker described, not a
// lifecycle outcome Orchestra decided.
var inc domain.ObservationIncident
if json.Unmarshal(e.Payload, &inc) != nil || inc.Signature == "" {
continue
}
o := base
o.Kind = domain.ObservationWorkerFailure
o.TaskID = inc.TaskID
o.Detail = inc.Detail
o.Repeats = inc.RepeatCount
add(domain.DebtOperational,
domain.DebtSignature(domain.DebtOperational, inc.Signature, inc.WorkerID, "worker"),
o, "workers report "+inc.Signature, review.Important)
case domain.EventOperatorInterventionRecorded:
var in domain.OperatorIntervention
if json.Unmarshal(e.Payload, &in) != nil || !in.Kind.Valid() {
continue
}
o := base
o.Kind = domain.ObservationManualIntervention
o.TaskID = in.TaskID
o.Detail = in.Reason
o.Paths = in.Components
add(domain.DebtOperational,
domain.DebtSignature(domain.DebtOperational, string(in.Kind), in.WorkerID, "manual"),
o, "an operator repairs this by hand ("+string(in.Kind)+")", review.Important)
case domain.EventPlanMismatchRecorded:
o := base
o.Kind = domain.ObservationPlanMismatch
@@ -167,16 +197,15 @@ func projectDebt(events []domain.Event, readArtifact func(string) ([]byte, error
// hole in the system. A kind the log could carry and does not is a fact about
// this history.
func debtGaps(seen map[domain.ObservationKind]bool) []domain.EvidenceGap {
gaps := []domain.EvidenceGap{
{Kind: domain.ObservationManualIntervention, Durable: false,
Reason: "no event type records an operator repair, so every manual recovery is invisible to this ledger"},
{Kind: domain.ObservationWorkerFailure, Durable: false,
Reason: "worker observations live in worker memory and reach the coordinator only inside WorkerHealth, which is not persisted"},
}
// Both of these were once permanent holes in the system. They are ordinary
// evidence now, so their absence is a fact about this history rather than
// about Orchestra.
var gaps []domain.EvidenceGap
for _, k := range []domain.ObservationKind{
domain.ObservationBlockReason, domain.ObservationFailureClass,
domain.ObservationReviewFinding, domain.ObservationPlanMismatch,
domain.ObservationDeferredFinding,
domain.ObservationDeferredFinding, domain.ObservationWorkerFailure,
domain.ObservationManualIntervention,
} {
if !seen[k] {
gaps = append(gaps, domain.EvidenceGap{Kind: k, Durable: true,
+83 -15
View File
@@ -64,24 +64,92 @@ func TestProjectDebtIgnoresOrdinaryLifecycleStops(t *testing.T) {
// not recorded anywhere".
func TestProjectDebtReportsWhatItCannotSee(t *testing.T) {
ledger := ProjectDebt(nil)
var manual, worker bool
for _, g := range ledger.Gaps {
if g.Durable {
continue
}
switch g.Kind {
case domain.ObservationManualIntervention:
manual = true
case domain.ObservationWorkerFailure:
worker = true
}
}
if !manual || !worker {
t.Fatalf("the two known holes must always be reported: %+v", ledger.Gaps)
}
for _, g := range ledger.Gaps {
if g.Reason == "" {
t.Fatalf("gap %q has no reason", g.Kind)
}
// Slice B closed the two holes this ledger used to report about
// itself. Every silence is now a fact about one history, never a kind
// of evidence the system cannot record at all.
if !g.Durable {
t.Fatalf("gap %q is reported as unrecordable: %+v", g.Kind, g)
}
}
var worker, manual bool
for _, g := range ledger.Gaps {
switch g.Kind {
case domain.ObservationWorkerFailure:
worker = true
case domain.ObservationManualIntervention:
manual = true
}
}
if !worker || !manual {
t.Fatalf("an empty history should still name both kinds as absent: %+v", ledger.Gaps)
}
}
// The whole point of incidents. One worker stuck in a retry loop must not
// manufacture recurrence, while its intensity is still on the record.
func TestRecurrenceCountsIncidentsAndKeepsIntensitySeparate(t *testing.T) {
closed := func(id, worker, task, epoch string, repeats int) domain.Event {
b, _ := json.Marshal(domain.ObservationIncident{
ID: id, WorkerID: worker, TaskID: task, LeaseEpoch: epoch,
Signature: "lease <id> not renewed: agent status idle and pane unchanged",
Detail: "lease " + task + " not renewed: agent status idle and pane unchanged",
RepeatCount: repeats, CloseReason: domain.ObservationCloseEpochChange,
})
return domain.Event{ID: id, Type: domain.EventObservationIncidentClosed, TaskID: "system", Payload: b}
}
ledger := ProjectDebt([]domain.Event{
closed("i1", "workpc-claude", "task-a", "e1", 301),
closed("i2", "workpc-claude", "task-b", "e2", 2),
})
if len(ledger.Items) != 1 {
t.Fatalf("one kind of failure produced %d items", len(ledger.Items))
}
item := ledger.Items[0]
if len(item.Observations) != 2 {
t.Fatalf("recurrence = %d, want one per incident", len(item.Observations))
}
intensity := 0
for _, o := range item.Observations {
if o.Kind != domain.ObservationWorkerFailure {
t.Fatalf("observation kind = %q", o.Kind)
}
intensity += o.Repeats
}
if intensity != 303 {
t.Fatalf("intensity = %d, want 303 carried alongside a recurrence of 2", intensity)
}
tasks := map[string]bool{}
for _, o := range item.Observations {
tasks[o.TaskID] = true
}
if len(tasks) != 2 {
t.Fatalf("the two incidents are not attributed to their tasks: %+v", item.Observations)
}
}
// A repair the operator made by hand is evidence like any other, once they say
// it happened.
func TestAnOperatorRepairBecomesDebtEvidence(t *testing.T) {
b, _ := json.Marshal(domain.OperatorIntervention{
WorkerID: "workpc-opencode", Kind: domain.InterventionTransactionClean,
Reason: "deleted a release transaction stuck at prepared so the pane could be reused",
})
ledger := ProjectDebt([]domain.Event{{
ID: "i1", Type: domain.EventOperatorInterventionRecorded, TaskID: "system", Payload: b,
}})
if len(ledger.Items) != 1 || len(ledger.Items[0].Observations) != 1 {
t.Fatalf("the repair produced no debt evidence: %+v", ledger.Items)
}
if got := ledger.Items[0].Observations[0].Kind; got != domain.ObservationManualIntervention {
t.Fatalf("kind = %q", got)
}
for _, g := range ledger.Gaps {
if g.Kind == domain.ObservationManualIntervention {
t.Fatal("manual intervention is still reported as missing from a history that contains one")
}
}
}
+84 -3
View File
@@ -49,6 +49,9 @@ type Store struct {
cursors map[string]string
cursorPath string
decisionSource map[string]string
// openObservations are the incidents opened and not yet closed, by id.
// Derived from the log, so a restart finds them again.
openObservations map[string]domain.ObservationIncident
// PreLease runs immediately before a lease is minted, which is the single
// point where ownership of a task begins. Reconciliation of newer human
// input belongs here rather than in any individual launch path, because a
@@ -96,7 +99,7 @@ func Open(dir string) (*Store, error) {
if t, ok := s.tasks[e.TaskID]; ok && e.Version != t.Version+1 {
return nil, domain.ErrConflict
}
global := e.Type == "QuotaReported" || e.Type == "StandupAdvisory" || e.Type == "ApprovalGranted" || e.Type == "ApprovalDenied"
global := domain.EventWithoutTask(e.Type)
if _, ok := s.tasks[e.TaskID]; !ok && e.Type != "TaskCreated" && !global {
return nil, domain.ErrNotFound
}
@@ -272,6 +275,9 @@ func (s *Store) apply(e domain.Event) error {
}
if p.Subject == domain.PlanPhaseSubject(rec.PlanRef, rec.PhaseID) {
t.PlanProgress.Phases[i].Status = domain.PlanPhaseVerified
// Record which tree the sign-off was about, so a later run
// at a different commit cannot inherit it (F63).
t.PlanProgress.Phases[i].ManualAtSHA = rec.AtSHA
t.Version = e.Version
s.replaceTask(e.TaskID, t)
}
@@ -309,6 +315,9 @@ func (s *Store) apply(e domain.Event) error {
}
t.PlanRef = p.ArtifactRef
t.PlanProgress = nil
// The replacement answers the contradiction. Carrying it
// further would put a solved problem in every later context.
t.PlanMismatch = nil
}
}
t.WorkPhase = p.Phase
@@ -484,6 +493,39 @@ func (s *Store) apply(e domain.Event) error {
t.Version = e.Version
s.replaceTask(e.TaskID, t)
return nil
case domain.EventObservationIncidentOpened:
// Open incidents are projected so a coordinator restart resumes them
// instead of orphaning them half-recorded. The event is what makes an
// incident durable at first sight; this is how it is found again.
var inc domain.ObservationIncident
if err := json.Unmarshal(e.Payload, &inc); err != nil {
return err
}
if s.openObservations == nil {
s.openObservations = map[string]domain.ObservationIncident{}
}
s.openObservations[inc.ID] = inc
return nil
case domain.EventObservationIncidentClosed:
var inc domain.ObservationIncident
if err := json.Unmarshal(e.Payload, &inc); err != nil {
return err
}
delete(s.openObservations, inc.ID)
return nil
case domain.EventPlanMismatchRecorded:
// Projected so the phase this reopens can be told what reopened it.
// The event is the record; this is the live instruction derived from
// it, and it stops being live as soon as a replacement seals or the
// stop is answered.
var m domain.PlanMismatch
if err := json.Unmarshal(e.Payload, &m); err != nil {
return err
}
t.PlanMismatch = &m
t.Version = e.Version
s.replaceTask(e.TaskID, t)
return nil
case domain.EventPlanPhaseVerified:
var pp domain.PlanPhaseRecord
if err := json.Unmarshal(e.Payload, &pp); err != nil {
@@ -585,8 +627,22 @@ func (s *Store) apply(e domain.Event) error {
// A question only stands while the task is blocked on it. Afterwards the
// answer is an ordinary standing decision and the log still holds the
// question, so keeping it on the task would put it in every later context.
//
// The blocker itself has the same lifetime and was outliving it (F67). A
// task resumed from a plan mismatch kept reporting block_reason
// plan_mismatch while it ran, so every surface reading the projection
// showed a stop that had already been answered.
if t.State != domain.StateBlocked {
t.DecisionRequest = nil
t.Blocker, t.BlockReason = "", ""
// The contradiction is cleared only by the correction that answers the
// stop, never by any other event that happens to find the task
// unblocked. This block runs for every event, and clearing it here
// unconditionally erased the contradiction at the very rotation the
// reopen causes, which is exactly when the planner needs it.
if e.Type == "TaskCorrected" {
t.PlanMismatch = nil
}
}
if phase, ok := p["lifecycle_phase"].(string); ok && phase != "" {
t.LifecyclePhase = phase
@@ -814,7 +870,7 @@ func (s *Store) Append(e domain.Event) error {
return fmt.Errorf("%w: corrects references unknown event %q for this task", domain.ErrInvalid, corrects)
}
}
global := e.Type == "QuotaReported" || e.Type == "StandupAdvisory" || e.Type == "ApprovalGranted" || e.Type == "ApprovalDenied"
global := domain.EventWithoutTask(e.Type)
if !taskExists && e.Type != "TaskCreated" && !global {
return domain.ErrNotFound
}
@@ -876,7 +932,7 @@ func (s *Store) Append(e domain.Event) error {
// for every lifecycle mutation.
func (s *Store) validateTransition(e domain.Event, t domain.Task, exists bool, p map[string]any) error {
if !exists {
if e.Type != "TaskCreated" && e.Type != "QuotaReported" && e.Type != "StandupAdvisory" && e.Type != "ApprovalGranted" && e.Type != "ApprovalDenied" {
if e.Type != "TaskCreated" && !domain.EventWithoutTask(e.Type) {
return domain.ErrNotFound
}
return nil
@@ -1080,6 +1136,31 @@ func (s *Store) Artifact(ref string) ([]byte, error) {
return b, nil
}
// NoteObservation updates an open incident's last actual occurrence. It writes
// no event: the aggregate is durable when the incident is finalized, and
// appending one per heartbeat is exactly the spam this design exists to avoid.
func (s *Store) NoteObservation(inc domain.ObservationIncident) {
s.mu.Lock()
defer s.mu.Unlock()
if _, ok := s.openObservations[inc.ID]; ok {
s.openObservations[inc.ID] = inc
}
}
// OpenObservations returns the incidents that are open, newest first by first
// sight. Open means not yet finalized as durable evidence, never "the failure
// is happening right now".
func (s *Store) OpenObservations() []domain.ObservationIncident {
s.mu.Lock()
defer s.mu.Unlock()
out := make([]domain.ObservationIncident, 0, len(s.openObservations))
for _, inc := range s.openObservations {
out = append(out, inc)
}
sort.Slice(out, func(i, j int) bool { return out[i].FirstSeen.After(out[j].FirstSeen) })
return out
}
func (s *Store) Task(id string) (domain.Task, bool) {
s.mu.Lock()
defer s.mu.Unlock()
+69
View File
@@ -723,3 +723,72 @@ func TestBlockRetainsLeaseEpochForALaterReopen(t *testing.T) {
t.Fatalf("last lease epoch %q, want %q", after.LastLeaseEpoch, epoch)
}
}
// F66's projection has to survive the rotation the reopen causes. The session
// that reported the contradiction hands off, a successor leases, and only then
// is the planning context rendered. Live on run 21, the field was gone by
// then: the reopen recorded it and the rotation lost it.
func TestTheContradictionSurvivesTheRotationItCauses(t *testing.T) {
s, err := Open(t.TempDir())
if err != nil {
t.Fatal(err)
}
b := []byte(`{"source":"s","external_id":"x","project":"p"}`)
if err := s.Append(domain.Event{ID: "create", Type: "TaskCreated", TaskID: "t", Version: 1, Payload: b, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
if _, err := s.Lease("t", "h", time.Minute); err != nil {
t.Fatal(err)
}
// Frame to implement, the shortest legal route to the phase a mismatch
// can be reported from.
task, _ := s.Task("t")
toImplement, _ := json.Marshal(map[string]any{"phase": "implement", "from": "frame"})
if err := s.Append(domain.Event{ID: "impl", Type: domain.EventWorkPhaseChanged, TaskID: "t", Version: task.Version + 1, Payload: toImplement, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
task, _ = s.Task("t")
m, _ := json.Marshal(map[string]any{
"plan_ref": "plan-a", "phase_id": "phase-2",
"at_sha": "0123456789012345678901234567890123456789",
"observed": "the body is assembled inline",
"contradicts": "the plan says one helper returns it",
"harness_id": task.Lease.HarnessID,
"lease_epoch": task.Lease.Epoch,
"requested_action": "replan",
})
if err := s.Append(domain.Event{ID: "mismatch", Type: domain.EventPlanMismatchRecorded, TaskID: "t", Version: task.Version + 1, Payload: m, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
if got, _ := s.Task("t"); got.PlanMismatch == nil {
t.Fatal("the contradiction was not projected at all")
}
// The reopen, then the rotation it causes.
task, _ = s.Task("t")
ph, _ := json.Marshal(map[string]any{"phase": "plan", "from": "implement", "reopen": domain.EventPlanMismatchRecorded, "reopen_phase_id": "phase-2"})
if err := s.Append(domain.Event{ID: "reopen", Type: domain.EventWorkPhaseChanged, TaskID: "t", Version: task.Version + 1, Payload: ph, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
ref, err := s.PutArtifact([]byte("handoff"))
if err != nil {
t.Fatal(err)
}
task, _ = s.Task("t")
rel, _ := json.Marshal(map[string]any{"handoff_ref": ref, "anchor_sha": "0123456789012345678901234567890123456789", "harness_id": task.Lease.HarnessID, "lease_epoch": task.Lease.Epoch, "expected_version": task.Version})
if err := s.Append(domain.Event{ID: "release", Type: "TaskReleased", TaskID: "t", Version: task.Version + 1, Payload: rel, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
if _, err := s.Lease("t", "h", time.Minute); err != nil {
t.Fatal(err)
}
// This is the moment the planning context is rendered.
got, _ := s.Task("t")
if got.PlanMismatch == nil {
t.Fatal("the planner is convened to settle a contradiction it is no longer told about")
}
if got.PlanMismatch.PhaseID != "phase-2" {
t.Fatalf("the contradiction changed across the rotation: %+v", got.PlanMismatch)
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

+635
View File
@@ -0,0 +1,635 @@
# orchestra web control ui
status: mockup specification
app: orchestra
shared design system: ethos
accent: signal violet `#8F7AE5`
motif: routing fork
primary operator: single infrastructure operator
## 1. purpose
orchestra's web ui is an operator console for understanding and steering agent work without reconstructing state from logs, tmux sessions, gitea, or the event stream.
it has three jobs:
1. show what needs the operator now.
2. show what orchestra believes and what an agent is actually doing.
3. expose intervention and forensic detail without making the default screens noisy.
it is not a generic project-management product, an analytics dashboard, or a chat-first interface.
## 2. ethos application
orchestra inherits the ethos shell and neutral system unchanged.
### fingerprint
- accent: signal violet `#8F7AE5`
- motif: routing fork
- accent use: active navigation, focus, selected phase, primary operator action, routing/transition marks
- motif use: app mark, phase transitions, routing state, empty states, one hero/detail moment
suggested token block:
```css
[data-app="orchestra"] {
--accent: #8F7AE5;
--accent-hi: #A291EC;
--accent-dim: rgba(143, 122, 229, 0.14);
--accent-line: rgba(143, 122, 229, 0.32);
--accent-glow: rgba(143, 122, 229, 0.22);
}
```
### type
- geist sans: labels, task names, explanations, buttons, headings, operator-authored prose
- geist mono: task ids, epochs, worker ids, pane ids, timestamps, percentages, context counts, revisions, paths, refs, sha values, lease durations
- machine values must never silently use sans for visual convenience
### surfaces
- page room: `--bg-0`
- normal panel: `--bg-1`
- selected/raised content: `--bg-2`
- hover: `--bg-3`
- borders: 1px `--line`, `--line-hi` only where selection or risk needs stronger separation
- no blur or glass
- accent never washes a whole content panel
### motion
all ui transitions use ethos mechanical motion: 130170 ms, `cubic-bezier(0.2, 0, 0, 1)`, no spring or bounce.
## 3. shared shell
### desktop
- 64px vertical icon rail
- 56px top bar
- main content fills remaining viewport
- global `cmd+k` command/search surface in the top bar
- right side of top bar shows only high-value machine state: orchestra revision, live state, current time
primary navigation:
1. dashboard
2. tasks
3. decisions
4. workers
5. projects
6. review
7. settings
terminal is entered from a task or worker and does not need permanent primary navigation.
### mobile
- rail becomes bottom navigation with no more than five primary items
- secondary pages move behind task/project drill-down or header menus
- machine readouts relocate into detail screens rather than disappearing
- terminal remains a dedicated full-screen view
## 4. information hierarchy
orchestra should visually distinguish five classes of information.
| class | meaning | treatment |
|---|---|---|
| operator authority | human decision, correction, constraint | high prominence, accent signal |
| verified orchestra state | lifecycle, lease, phase, review, refs | neutral surfaces, machine values in mono |
| agent claims | proposed plan, handoff text, self-reported risk | visibly labeled as agent-supplied |
| historical/superseded | stale plan, previous handoff, old decision | muted, never visually dominant |
| fault/attention | blocked, refused, stale, mismatch | status color + explicit reason |
raw events and implementation details are drill-down evidence, not default-page content.
## 5. dashboard
mockup: `final/01-dashboard.png`
purpose: answer "what needs me, what is running, and is the system healthy?"
### layout
three primary regions only:
1. **needs your attention**
- trajectory gate
- blocking human decision
- pr awaiting review
- closed-without-merge/operator-required state
- one line explaining why each item needs the operator
2. **running tasks**
- task name/id
- current phase
- worker/harness
- last real progress
- context occupancy
- lease timing
- compact routing-fork phase indicator
3. **system state and capacity**
- workers online/available
- quota status
- active leases
- router queue/rejection state
- source reconciliation
- herdr health
### rules
- no disk, cpu, ram, or generic host-monitor metrics unless they directly block orchestration
- no vanity charts
- health should show actual counts/times, not vague "running" indicators
- dashboard actions should navigate to the relevant task/review/worker rather than becoming an operator-control surface itself
## 6. tasks list
mockup: `final/04-tasks.png`
purpose: browse all work under orchestra and locate a task quickly.
### groups
- needs attention
- running
- waiting/queued
- completed
- failed
### row fields
- task title/id
- project
- current phase
- compact routing-fork phase path
- worker/harness when assigned
- last update/progress
- context occupancy for active sessions
- status
### filters
- project
- phase
- worker/harness
- status
- attention state
avoid turning this into a kanban board. orchestra's lifecycle is stateful and event-backed; the list should expose state, not encourage arbitrary drag-and-drop mutation.
## 7. task detail
mockup: `final/02-task-detail.png`
purpose: provide one calm page answering "what are we doing?", "what is happening?", and "where are we in the workflow?"
### top area
- title
- task id
- project
- current phase
- steer/correct primary action
- secondary actions menu
### what we're doing
show only:
- goal
- acceptance summary
- latest effective human decisions when present
- active constraints when present
full contract, all decisions, and historical versions open in drill-down views.
### live execution
show only:
- worker/harness
- pane
- lease remaining
- context occupancy
- last progress
- one-line current activity
links:
- open live pane
- worker logs
- launch context
- git diff
### workflow
render the ace-fca path visibly:
`frame → research → plan → implement → review → pr → merged`
completed phases are quiet green, current phase uses the orchestra accent, future phases remain neutral.
phase detail, artifacts, refusal history, and raw events belong in drill-downs rather than the default task screen.
## 8. live pane / terminal
mockup: `final/03-terminal-live-pane.png`
purpose: show the actual tmux/herdr pane, not a reconstructed log view.
### behavior
- read-only by default
- exact pane dimensions and terminal rendering preserved when possible
- fit mode may scale visually but must not alter the underlying pane
- scrollback inspection must not disturb the live cursor
- `take control` explicitly enables keyboard forwarding
- strong visual state when control is captured
- escape or a fixed shortcut releases control
### why this screen exists
it must reveal real tui state such as:
- input still sitting in the editor
- queued input
- paste placeholders
- permission/approval dialogs
- `/clear`
- `@HANDOFF.md`
- agent busy/idle presentation
- tmux status line and pane identity
### surrounding chrome
keep it minimal:
- worker/pane identity
- read-only/control state
- fit/copy/scrollback controls
- lease epoch/timing
- context occupancy
- last verified progress
- pane/worker health
- small live event strip for orchestra-originated interactions and confirmations
when a pane is lost, preserve the last captured frame and state the exact loss time and lease consequence.
## 9. decisions list
mockup: `final/05-decisions.png`
purpose: inspect durable human authority and approvals across tasks.
this is not the human attention queue; that is review.
### list fields
- decision id
- short decision text
- task
- phase
- source/provenance
- requested/recorded time
- status: active, superseded, waiting, approved, rejected, auto
- operator/actor
### filters
- active/superseded
- kind: decision, correction, constraint, answer, scope change
- project
- task
- actor/source
### detail
opening a decision should show:
- exact value
- subject
- provenance
- supersedes/superseded-by links
- event sequence
- effective authority impact
machine provenance is mono; the human decision text remains sans.
## 10. workers
mockup: `final/06-workers.png`
purpose: answer "where can work run and is that execution machinery usable?"
### worker list fields
- worker/harness identity
- online/idle/busy/offline/degraded
- harness type
- current task when leased
- phase
- lease or heartbeat age
intentionally omit generic infrastructure metrics such as disk, memory, cpu, and uptime unless orchestra directly uses them for eligibility.
### worker detail
- current session/task
- pane id
- lease epoch/remaining
- last verified progress
- worker revision
- herdr status
- last heartbeat
- projects declared by the worker
- declared capabilities if orchestra actually consumes them
### capabilities
capabilities must come from worker registration/project configuration, never be guessed from host inspection.
the ui labels them **declared capabilities** and can expose registration provenance.
### registration
registration is worker-driven:
1. worker starts with coordinator credentials/configuration
2. worker announces identity, revision, harnesses, projects, capacity and declared capabilities
3. coordinator records it and heartbeat updates liveness
`add worker` should therefore be a registration guide/bootstrap flow, not a magical browser-side creation of a remote process.
## 11. projects
mockup: `final/07-projects.png`
purpose: group long-running repository/workflow configuration and its active work.
### project list
- project name
- short description
- state
- active/total task count
- attention/decision count
- last meaningful activity
avoid aggregate artifact counts or infrastructure metrics unless they help choose an action.
### project detail
- description
- source/repository identity
- configured worker/harness affinity
- workflow phases
- trajectory-gate policy
- quality gate
- human reconcile source
- safe-operation policy
- active tasks
- recent meaningful events
configuration edits belong in the project detail, while global defaults remain in settings.
## 12. review
mockup: `final/08-review.png`
purpose: the operator inbox: **what needs me to look at something and judge?**
workers show machinery. review shows human attention.
### review item types
- trajectory decision
- human blocker / decision request
- code/pr review
- ai review requiring operator override
- operator-required lifecycle state
- closed pr without merge
- information/fyi, optionally hidden by default
### list fields
- type
- task/item
- why it is here
- project
- priority
- age
worker/pane is secondary metadata, not a primary column.
### detail drawer/page
show the minimum evidence needed to act:
- why this needs the operator
- effective task goal/constraints relevant to the decision
- proposed options when applicable
- relevant plan/diff/review evidence
- provenance and age
- comments/context drill-down
primary action names must match the lifecycle action: make decision, answer blocker, review pr, request changes, retry, etc.
snooze/defer may exist for attention management but must not silently mutate underlying task truth.
## 13. settings
mockup: `final/09-settings.png`
purpose: configure orchestra itself, project-independent defaults, integrations, policies, and operator/security settings.
### tabs
1. general
2. appearance
3. notifications
4. access & security
5. integrations
6. agents & tools
7. policies
8. advanced
### general
- instance name/environment
- locale/time display
- default task preferences
- default workflow policy
- event/artifact retention
- storage/backup state only if orchestra owns it
### appearance
- theme: system/dark/light
- ethos remains fixed; no arbitrary shell redesign controls
- accessibility and reduced motion
### access & security
- surface tokens/status
- trusted review actors
- ignored actors
- operator sessions
- agent-surface auth
- secret/config source status without exposing secret values
### integrations
- gitea
- vikunja
- notification sinks
- source reconciliation state
show endpoint/identity and last successful interaction in mono.
### agents & tools
- harness definitions
- adapter/runtime configuration
- tmux/herdr ownership
- default launch behavior
- context occupancy thresholds
### policies
- work phase defaults
- trajectory gate
- bounded decision request budget
- retry policy
- reconcile failure handoff threshold
- quality/review/submission policy
### advanced
- event-store diagnostics
- snapshots/replay
- raw configuration
- migration/status information
- destructive operator actions in a visually separated danger zone
## 14. global command surface
`cmd+k` is the shared ethos command surface and should handle navigation plus safe operator actions.
examples:
- open task by id/title
- open worker/pane
- show decisions for task
- steer/correct active task
- answer current blocker
- retry failed task
- open review item
high-risk lifecycle actions require explicit confirmation and should not be the first fuzzy-search result for ordinary text.
## 15. status grammar
status is compact and consistent across screens.
recommended semantic colors:
- green: verified healthy/completed/accepted
- violet: current/selected/orchestra-controlled active state
- amber: waiting/degraded/operator attention
- red: failed/refused/danger
- gray: idle/unknown/historical
color never carries the entire meaning; every state also has text/iconography.
## 16. evidence drill-down
any important derived claim should be traceable without ssh.
from relevant detail screens the operator should be able to reach:
- event(s) that established the state
- exact generated launch context
- sealed research/plan/review/submission artifact refs
- git sha/ref
- worker journal observations
- input confirmation receipt
- pane capture/terminal
- source comment/review provenance
raw evidence is accessible but not visible by default.
## 17. copy rules
- labels describe what the operator controls, not implementation class names
- machine values remain exact and honest
- errors state what failed and what consequence follows
- no "something went wrong"
- no spinner where a queue depth, heartbeat age, poll timestamp, retry time, or progress value exists
examples:
- `source reconcile failed · retry at 14:06:00`
- `worker heartbeat 42s old · ineligible after 60s`
- `phase request refused · next phase is research`
- `pane lost at 13:42:18 · lease expires in 09:14`
## 18. responsive behavior
### tablet/mobile priorities
keep visible:
1. attention state
2. task title/phase
3. latest human authority
4. live execution state
5. primary action
collapse or drill down:
- long acceptance lists
- complete workflow history
- detailed lease metadata
- worker journal
- raw events
- project configuration
terminal becomes its own full-screen route and should not be squeezed into a card.
## 19. implementation order
recommended web-ui build order:
1. ethos shell + orchestra tokens/motif
2. dashboard
3. tasks list
4. task detail
5. live pane
6. review inbox
7. decisions list/detail
8. workers
9. projects
10. settings
11. global command surface
12. evidence/raw-event drill-down
build each screen against real orchestra api data rather than static ui-only state as early as practical.
## 20. verification
for every screen:
- render and visually inspect pixels
- check desktop and <=640px mobile layout
- assert no horizontal overflow
- verify mono/sans split visibly
- verify accent is signal-only
- verify no blur/backdrop-filter exists
- verify empty/error/loading states show real machine state
- compare implemented screen against the corresponding mockup as direction, not pixel-perfect contract
mockups communicate hierarchy, density, and interaction intent. orchestra state and the ethos design laws are the actual specification.