Compare commits
25 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| f25964c18e | |||
| 0aefe021b0 | |||
| 982741fe97 | |||
| 42c5f07844 | |||
| 79d20534b5 | |||
| 74cad5d374 | |||
| e0601296e0 | |||
| 438c1d6df3 | |||
| 76da8c40b7 | |||
| 575e3ef87e | |||
| 75396963ef | |||
| edf00761fd | |||
| 1330ad9943 | |||
| cb80dd1952 | |||
| de18f372d3 | |||
| bdc0d4d5be | |||
| 6cb2f932d8 | |||
| b5c37f693b | |||
| bbf3335857 | |||
| 99b209ba10 | |||
| ab3258833d | |||
| c587f2cc8d | |||
| 41658aea5b | |||
| 765bf2afc6 | |||
| cbd6b11c49 |
@@ -2527,3 +2527,492 @@ transaction and no session behind, on either worker.
|
||||
|
||||
Both workers hold no sessions and no release transactions. Deployed pair is
|
||||
still `3c7cf95` on both halves.
|
||||
|
||||
## Run 16, 2026-08-29: the rotation rung, half proven
|
||||
|
||||
Task `06G4SWEVP71FYYKAV5FV0ZK5ZG` on `34f3c28`, a three-phase plan
|
||||
(`78d25730140c`) with automated checks inside policy.
|
||||
|
||||
### Proven: a successor inherits the whole sealed plan
|
||||
|
||||
The launch context of the session that picked the task up after its rotation
|
||||
carries the complete plan: overview, current state, desired end state,
|
||||
non-goals, approach, all three phases with their files, changes and
|
||||
verification, testing strategy, risks, migration, and ten research citations.
|
||||
|
||||
### Not a defect: that successor was not given phase progress
|
||||
|
||||
`renderPlanProgress` runs only for `WorkPhaseImplement`
|
||||
(`internal/agentctx/agentctx.go:321`). This successor picked up in review, and
|
||||
the code says why: an independent review must reconstruct the change from the
|
||||
diff rather than inherit the implementer's account. Progress is withheld there
|
||||
deliberately.
|
||||
|
||||
So the rung still needs an **implement-phase** successor. The records exist and
|
||||
are durable, projected by `Task.PlanPhases()`.
|
||||
|
||||
### Why the mid-implement rotation missed
|
||||
|
||||
A trivial task spends about four minutes in implement, and all three phases
|
||||
verified inside two and a half of them, every one against the same tree
|
||||
`3da86dbf8ad5`. The implementer does the whole change first, then verifies each
|
||||
phase. The suspend, edit and restart that sets `handoff_requested` takes about
|
||||
a minute, and it landed after the implementer had already asked for review.
|
||||
|
||||
Two mechanics worth keeping:
|
||||
|
||||
- **The web handoff action cannot drive this.** `RequestHandoff` needs a local
|
||||
coordinator and every session here is worker-owned, so it answers 503. That
|
||||
is the Design A guardrail working, and a fourth UI-exposed gap.
|
||||
- **A state-file edit does not survive a running worker.** It holds sessions in
|
||||
memory and writes them back. Suspend, edit, then restart; resuming lets the
|
||||
old copy win. This cost one attempt to learn.
|
||||
|
||||
### F62: a requested handoff that is never answered is invisible
|
||||
|
||||
The rotation is agent-driven: the worker asks, and the agent must write
|
||||
`HANDOFF.md` at a turn boundary. The review agent never did. Renewals stopped,
|
||||
the lease expired at 10:12:05, and the task requeued having lost an attempt.
|
||||
|
||||
Worker health recorded exactly one thing, 34 times:
|
||||
|
||||
```text
|
||||
x34 lease 06G4SWEVP71FYYKAV5FV0ZK5ZG not renewed: agent status idle and pane unchanged
|
||||
```
|
||||
|
||||
Nothing says a handoff was requested and left unanswered. There is no timeout,
|
||||
no retry, and no observation. The only visible consequence is an expiry that
|
||||
looks identical to an ordinary idle one. Diagnosing it needed the worker state
|
||||
file.
|
||||
|
||||
The ring earned its keep again: one distinct message with a count of 34, rather
|
||||
than 34 overwrites of one slot.
|
||||
|
||||
**Fixed, not yet proven live.** The request is now stamped
|
||||
(`herdr.Session.HandoffRequestedAt`) and the wait is bounded by
|
||||
`watchHandoff` in the worker:
|
||||
|
||||
- the lease renews while Orchestra is explicitly waiting, because a quiet pane
|
||||
is the answer the agent was asked for — the renewal gate's new case, bounded
|
||||
by `handoffAnswerTimeout`;
|
||||
- the request is re-sent once at `handoffRetryAfter` (4 minutes), with the same
|
||||
reason it was first asked with;
|
||||
- at 10 minutes the worker nacks with failure class `handoff_unanswered`, and
|
||||
the coordinator emits `TaskReleased reason=handoff_unanswered` rather than
|
||||
letting the lease die as generic idleness.
|
||||
|
||||
`DebtClassForFailureClass` knows the class, so a harness that repeatedly
|
||||
ignores handoff requests now accumulates in the debt ledger instead of hiding
|
||||
inside `lease_expired`.
|
||||
|
||||
Runtime proof still owed: force a request the agent will not answer, and read
|
||||
the release event rather than the worker state file.
|
||||
|
||||
## Run 17, 2026-08-29: F62 proven live on `c587f2c`
|
||||
|
||||
Task `06G4V20T528ZTER7KZBVGNAYXC`, a five-phase plan
|
||||
(`3b16b77054b9`) on the deployed pair, coordinator and worker both at
|
||||
`c587f2c`.
|
||||
|
||||
### The trigger was production, not a rig
|
||||
|
||||
The rig planned for this run was a state-file lever. It was never needed. The
|
||||
`implement -> review` phase change asked for a handoff through the ordinary
|
||||
path (`rotateForPhase`, reason `phase_changed`), and the agent ignored it
|
||||
because this task's brief instructed it to from phase 4 onward. Nothing touched
|
||||
tmux, no process was stopped, and the pane stayed open for the whole wait.
|
||||
|
||||
```text
|
||||
12:48:09 implement -> review, handoff requested, reason=phase_changed
|
||||
12:50:09 TaskLeaseRenewed agent idle, pane capture unchanged
|
||||
12:52:15 pane input confirmed the single resend, 4m06s after the request
|
||||
12:52:39 TaskLeaseRenewed
|
||||
12:55:14 TaskLeaseRenewed
|
||||
12:57:49 TaskLeaseRenewed
|
||||
12:58:14 TaskReleased reason=handoff_unanswered failure_class=handoff_unanswered
|
||||
last_error="handoff requested (phase_changed) and unanswered for 10m5s"
|
||||
12:59:14 TaskLeased successor leases normally
|
||||
13:02:20 TaskSubmitted
|
||||
```
|
||||
|
||||
Every assertion the operator set for this rung held:
|
||||
|
||||
- the release does not look like an ordinary `lease_expired`: it carries
|
||||
`reason=handoff_unanswered` and `lifecycle_phase=handoff_unanswered`;
|
||||
- the idle-renewal rule did not kill the wait: four renewals on an idle pane
|
||||
with no capture movement, each one the gate would have refused before this
|
||||
commit;
|
||||
- the resend preserved the reason the request was first made with;
|
||||
- exactly one resend, not one per tick;
|
||||
- the timeout produced the class at `10m5s`, one tick past the bound;
|
||||
- nothing stale was left: `releases: {}`, no session, no lease, and zero tmux
|
||||
panes matching the id;
|
||||
- the next attempt leased 60s later and reached `TaskSubmitted`.
|
||||
|
||||
The debt projection classified it with no new code beyond the class mapping:
|
||||
|
||||
```text
|
||||
v1:operational:handoff_unanswered:workpc-claude:lease operational important
|
||||
"handoff requested (phase_changed) and unanswered for 10m5s"
|
||||
```
|
||||
|
||||
**Inference, not observation.** The resent prompt's text was not captured. That
|
||||
it carried the `phase_changed` wording rests on the code path and the timing,
|
||||
not on a pane capture. A future rig should publish a capture across the resend.
|
||||
|
||||
### The implement-successor rung missed again, and why
|
||||
|
||||
A five-phase plan does not lengthen the implement phase. This one ran 12:45:18
|
||||
to 12:48:09, about 2m50s, with verifications 20 to 40 seconds apart:
|
||||
|
||||
```text
|
||||
12:45:47 phase-1 12:46:07 phase-2 12:46:32 phase-3
|
||||
12:47:12 phase-4 12:47:52 phase-5 12:48:09 implement -> review
|
||||
```
|
||||
|
||||
The brief's explicit one-phase-at-a-time requirement was followed, and it
|
||||
still took under three minutes. Phase count is not the lever; volume of
|
||||
mechanical work is.
|
||||
|
||||
`awaiting_manual_verification` does not hold the phase open either. All five
|
||||
phases carried that status and the agent moved on regardless.
|
||||
|
||||
### The state-file lever cannot be driven with `restart` alone
|
||||
|
||||
The documented procedure is stop, edit, start. The unattended sudo rule grants
|
||||
`install` and `systemctl restart orchestra-worker`, with no `stop` or `start`.
|
||||
With `restart` only, the running worker's five-second tick save landed between
|
||||
the edit and the restart, and the edit was gone. Widening the sudoers line is
|
||||
the fix.
|
||||
|
||||
### One cosmetic note
|
||||
|
||||
`Session.HandoffRequestedAt` serializes as `0001-01-01T00:00:00Z` when unset,
|
||||
because `omitempty` does not omit a zero `time.Time`. The worker parses it back
|
||||
to a zero value and `watchHandoff` starts the clock, so behaviour is correct.
|
||||
It is the same marshalling trap the operator console hit.
|
||||
|
||||
## Run 18, 2026-08-29: the implement-successor rung, proven
|
||||
|
||||
Task `06G4VF5HZW7Q4JBM3TTY7W1Y64` on `c587f2c`, a five-phase plan whose phases
|
||||
are ten named checks rather than one refactor. Volume of mechanical work is
|
||||
what keeps the implement phase open; phase count does not.
|
||||
|
||||
### The rotation
|
||||
|
||||
```text
|
||||
13:43:10 implement session launched
|
||||
13:43:44 phase-1 verified
|
||||
13:44:04 phase-2 verified
|
||||
13:44:35 handoff requested, reason=milestone, delivered by watchHandoff's resend
|
||||
13:45:25 TaskLeaseRenewed
|
||||
13:46:51 TaskReleased handoff_ref 0b4fae6f705a anchor 8ea3c7d1ae8f
|
||||
13:46:51 TaskLeased successor, same harness
|
||||
13:46:55 TaskPickupValidated
|
||||
13:47:06 TaskLaunchAcknowledged, still in implement
|
||||
```
|
||||
|
||||
The lever was the worker state file, backdated past `handoffRetryAfter` so the
|
||||
request went out on the next tick. Under `c587f2c` that lever now produces a
|
||||
real prompt: the flag alone used to sit there unasked, which is how run 16
|
||||
ended in a silent expiry.
|
||||
|
||||
### What the successor was given
|
||||
|
||||
From `.orchestra/launch.md`, written at 13:47:05:
|
||||
|
||||
```text
|
||||
## Current phase
|
||||
implement: Implement the accepted plan below. Verify as you go.
|
||||
|
||||
## Verified git state
|
||||
- head: 8ea3c7d1ae8fd960897344694645117adbd8ee82
|
||||
- uncommitted changes: false
|
||||
|
||||
## Current human decisions
|
||||
None recorded. Work from the goal and acceptance above.
|
||||
|
||||
## Plan progress
|
||||
Orchestra established this by running the plan's own verification. You cannot
|
||||
write it.
|
||||
- phase-1 (...): automated checks passed at d3acd4989a20, stale because the
|
||||
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
|
||||
- phase-2 (...): automated checks passed at d3acd4989a20, stale because the
|
||||
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
|
||||
- phase-3 (Checks 7 to 8): not started
|
||||
- phase-4 (...): not started
|
||||
- phase-5 (...): not started
|
||||
```
|
||||
|
||||
Everything the rung asked for is there: the complete accepted plan with all
|
||||
five phases, their files, changes and verification; the phases already
|
||||
verified; the first unfinished phase; and the current human authority. The
|
||||
launch context also carries the accepted research and its dead ends.
|
||||
|
||||
**SHA staleness renders itself.** Neither phase is reported as simply passed.
|
||||
Each says the checks passed at `d3acd4989a20` and are stale because the tree
|
||||
has moved to `8ea3c7d1ae8f`. That is half of the manual-verification rung
|
||||
observed without being asked for.
|
||||
|
||||
### The lever still loses a race, sometimes
|
||||
|
||||
`systemctl restart` alone leaves a five-second window in which the running
|
||||
worker's tick save can clobber the edit. It clobbered run 17's attempt and
|
||||
survived run 18's. Stop, edit, start is the reliable sequence, and it needs
|
||||
`systemctl stop` and `start` in the unattended sudo rule.
|
||||
|
||||
## Run 19, 2026-08-29: the manual verification seam, and F63
|
||||
|
||||
Two tasks, both ingested from Gitea so that comments reconcile:
|
||||
`06G4W2TMZ9MW9SEBM5MGTM5288` (issue 24, plan `500464b9`) and
|
||||
`06G4W63545T3RV8SSKGKN11B3G` (issue 26, plan `53f7918b`).
|
||||
|
||||
A burn-in task cannot test this rung. Comments reconcile only for the source
|
||||
the task came from, so a task created with `source: "burnin"` has no comment
|
||||
stream at all.
|
||||
|
||||
### Proven: the keyed seam
|
||||
|
||||
```text
|
||||
15:09:15 PlanPhaseVerified phase-1 awaiting_manual_verification at 80728967
|
||||
comment: "Looks good to me. Nice work on this one, ship it."
|
||||
17:10:03 HumanDecisionRecorded subject=operator_instruction
|
||||
phase-1 unchanged, still awaiting_manual_verification
|
||||
comment: "orchestra verify phase-1"
|
||||
17:10:25 HumanDecisionRecorded subject=plan_phase_verification:500464b9…:phase-1
|
||||
phase-1 verified
|
||||
```
|
||||
|
||||
All five phases reached `awaiting_manual_verification` on their automated
|
||||
checks alone. A generic approval satisfied nothing. The keyed form flipped
|
||||
exactly the phase it named, under the plan ref Orchestra supplied rather than
|
||||
one the commenter chose.
|
||||
|
||||
### Proven: re-verification at a new SHA
|
||||
|
||||
On issue 26, inside one live implement lease, with the worker stopped for the
|
||||
whole edit:
|
||||
|
||||
```text
|
||||
15:23:00 PlanPhaseVerified phase-1 awaiting_manual_verification at f5d8f90b
|
||||
operator commit moves the tree f5d8f90b -> 1f07b89c
|
||||
.orchestra/plan-progress.json requests phase-1 again
|
||||
15:23:17 PlanPhaseVerified phase-1 awaiting_manual_verification at 1f07b89c
|
||||
17:23:16 HumanDecisionRecorded subject=plan_phase_verification:53f7918b…:phase-1
|
||||
phase-1 verified at 1f07b89c
|
||||
```
|
||||
|
||||
The rerun ran the plan's own commands at the new tree and wrote a fresh record
|
||||
bound to it. The worker never reads a command out of the request: the request
|
||||
names a phase and nothing else about it is trusted.
|
||||
|
||||
### F63: a manual sign-off outlives the tree it was given against
|
||||
|
||||
The second half of the same rig, with no new comment:
|
||||
|
||||
```text
|
||||
operator commit moves the tree 1f07b89c -> 28af84ff
|
||||
.orchestra/plan-progress.json requests phase-1 again
|
||||
15:24:02 PlanPhaseVerified phase-1 verified at 28af84ff
|
||||
```
|
||||
|
||||
`RecordPlanPhaseVerification` consults `manuallySignedOff`
|
||||
(`internal/operations/planprogress.go:127,152`), which asks only whether a
|
||||
decision with that subject exists. It does, forever, for that plan ref and
|
||||
phase id. So the automated half of a phase is re-established at every new SHA
|
||||
while the manual half is asserted from a tree the human never saw.
|
||||
|
||||
This is the same class of bug `AtSHA` exists to prevent, applied to the half
|
||||
that `AtSHA` does not cover. A manual check on this project is a human reading
|
||||
printed output; an edit between the sign-off and the rerun can change exactly
|
||||
that output.
|
||||
|
||||
Only a replan clears it, because `PlanPhases()` drops records whose plan ref is
|
||||
not the accepted one.
|
||||
|
||||
Proposed shape, not built: when the reducer flips a record on a keyed decision
|
||||
(`internal/store/store.go:273`), it knows the record it satisfied and that
|
||||
record's `AtSHA`. Stamp that sha on the decision, and have `manuallySignedOff`
|
||||
require it to match the run being recorded. A sign-off that arrives before any
|
||||
run has no sha and keeps today's behaviour, so the ordinary ordering is
|
||||
unchanged.
|
||||
|
||||
### Also established
|
||||
|
||||
- `review -> implement` is legal in `legalPhaseTransitions`, but
|
||||
`AdvanceWorkPhase` only walks the project's path forward. A task that has
|
||||
left implement can return only through a review that requests changes, not
|
||||
through the operator phase endpoint.
|
||||
- The agent's re-verification request is a file an operator can write:
|
||||
`.orchestra/plan-progress.json`, `{"phase": "...", "status":
|
||||
"ready_for_verification"}`. It is the whole lever for this rung.
|
||||
- Ingest turns every issue label into a required capability, so a rig issue
|
||||
must carry no labels or it will not schedule.
|
||||
|
||||
## Run 20, 2026-08-29: the mismatch and replan ladder, and three defects
|
||||
|
||||
Task `06G4WJ9T4F35NZC4Z8QQXM9Z6G` (issue 29) on `de18f37`, plus a first attempt
|
||||
on `06G4WDJ06G7FK00QF73Z9DDBT4` (issue 27) on `c587f2c` that found F65.
|
||||
|
||||
The lever throughout is the agent's own file, written by the operator with the
|
||||
worker stopped: `.orchestra/plan-mismatch.json`, carrying `phase_id`,
|
||||
`observed`, `contradicts`, `evidence` and `requested_action`. The worker fills
|
||||
`plan_ref` and `at_sha` from what it can verify, never from the file.
|
||||
|
||||
### F65: a task that stops for a human never stopped
|
||||
|
||||
Found on the first attempt, live:
|
||||
|
||||
```text
|
||||
16:04:25 PlanMismatchRecorded phase-2 human_decision
|
||||
16:04:25 worker: plan mismatch refused: task version conflict
|
||||
16:04:48 PlanPhaseVerified phase-1 work continues as if nothing happened
|
||||
```
|
||||
|
||||
`Store.Append` fences every lifecycle event on a leased task against the
|
||||
current owner and epoch. `blockForPlanMismatch` carried neither `harness_id`
|
||||
nor `lease_epoch`, so the block was refused on every task that was actually
|
||||
running, which is every task that can produce a mismatch. The observation was
|
||||
durable and the transition silently did not happen: the repo's own documented
|
||||
failure shape. `raiseTrajectoryGate` had the identical omission, so the
|
||||
trajectory gate could never have stopped a live task either.
|
||||
|
||||
`RecordPlanMismatch` also returns that error after the record is already
|
||||
appended, so the worker told the agent its report was refused for a report
|
||||
Orchestra had accepted.
|
||||
|
||||
Fixed in `de18f37` with one `fenceToLease` helper, called by all three stops.
|
||||
The human-decision path already did it correctly and explained why in a
|
||||
comment; that comment is now the helper.
|
||||
|
||||
**Why no test caught it.** `planWith` never leased its task, so every
|
||||
plan-progress and plan-mismatch test ran in a state no agent can be in. The
|
||||
`lease` helper in the same package carries a comment warning about exactly
|
||||
that. It leases now, and the mismatch block test fails without the fence with
|
||||
the same `task version conflict`.
|
||||
|
||||
### Proven: human_decision stops, and the reply resumes the same plan
|
||||
|
||||
```text
|
||||
16:24:21 PlanMismatchRecorded phase-2 human_decision
|
||||
16:24:21 TaskBlocked plan_mismatch, lifecycle awaiting_human
|
||||
18:24:46 HumanDecisionRecorded operator_instruction (Gitea's timestamp)
|
||||
16:25:16 TaskCorrected resumed
|
||||
16:25:16 TaskLeased plan_ref unchanged at 0bc8ef84
|
||||
16:25:21 TaskPickupValidated
|
||||
```
|
||||
|
||||
55 seconds from block to resumed, on the same plan, with no reseal. The
|
||||
blocker packet states the observation, the plan's claim and the evidence.
|
||||
|
||||
This is also the live proof of F64: before `6cb2f93` nothing read a blocked
|
||||
task's comments, because the reconciler ran only before a lease and at a turn
|
||||
boundary, and a blocked task is never leased.
|
||||
|
||||
### Proven: replan keeps the old plan until a replacement seals
|
||||
|
||||
```text
|
||||
16:26:07 PlanMismatchRecorded phase-3 replan
|
||||
16:26:07 WorkPhaseChanged plan reopen=PlanMismatchRecorded
|
||||
during planning: plan_ref 0bc8ef84 accepted, phase-1 and phase-2 counted
|
||||
after the seal:
|
||||
plan_ref 922f293b
|
||||
plan_history [0bc8ef84]
|
||||
progress empty
|
||||
successor in implement: four phases, all "not started"
|
||||
```
|
||||
|
||||
The successor's launch context carries the replacement whole, with files,
|
||||
changes and verification per phase. The old progress stops counting because
|
||||
`PlanPhases()` drops records whose plan ref is not the accepted one. An
|
||||
abandoned replan would have cost nothing.
|
||||
|
||||
### F66: the planner replans blind
|
||||
|
||||
The planning session convened by a replan is given the ordinary plan brief and
|
||||
an empty template. Its launch context says nothing about the plan that already
|
||||
exists, the contradiction that reopened the phase, the phase id, the evidence,
|
||||
or the fact that its output supersedes an accepted plan. The only human input
|
||||
it carried was an unrelated earlier correction.
|
||||
|
||||
Everything it needs is already durable. `PlanMismatchRecorded` holds the phase,
|
||||
the observation, the contradiction and the evidence, and `reopenPhase` writes
|
||||
`reopen_phase_id` onto the `WorkPhaseChanged` payload. None of it reaches the
|
||||
agent, so nothing stops the replacement from sealing with the same
|
||||
contradiction in it.
|
||||
|
||||
### Smaller observation
|
||||
|
||||
The implement session kept working after the reopen. It verified phase-2
|
||||
against the old plan 40 seconds after the task had moved to `plan`. Harmless,
|
||||
because supersession discards it, but the reopen does not end the session that
|
||||
reported the contradiction. That session rotates at its own next boundary.
|
||||
|
||||
## Run 21, 2026-08-29: F66 proven, in two rounds
|
||||
|
||||
Tasks `06G4WW6TND26M16CZA6WE5T458` (issue 32, on `edf0076`) and
|
||||
`06G4XAFH1MBPC35VSJN7V3NS14` (issue 34, on `7539696`).
|
||||
|
||||
### Round one failed, and the failure was in the fix
|
||||
|
||||
`edf0076` projected the contradiction onto the task and rendered it into the
|
||||
plan-phase context. The projection appeared at the reopen and was gone before
|
||||
the planning session launched:
|
||||
|
||||
```text
|
||||
16:58:01 PlanMismatchRecorded phase-2 replan, plan reopened
|
||||
projection carries the contradiction
|
||||
16:58:20 TaskReleased the rotation the reopen causes
|
||||
projection empty
|
||||
16:59:29 replacement sealed, planner never told anything
|
||||
```
|
||||
|
||||
The clearing rule added with F67 sits **below** the reducer switch, so it runs
|
||||
for every event rather than for the correction it was written for. A release
|
||||
found the task unblocked and erased the contradiction. Two mistakes made it:
|
||||
the rule was written as if it were inside `case "TaskCorrected"`, which is
|
||||
merely the nearest case above it, and the unit test read the projection at the
|
||||
moment it was written rather than at the moment the planner reads it.
|
||||
|
||||
`7539696` scopes the clear to `TaskCorrected`, and the store test now walks the
|
||||
real sequence: mismatch, reopen, release, lease, then assert.
|
||||
|
||||
### Round two: the planner is told what it was convened to fix
|
||||
|
||||
```text
|
||||
18:00:17 PlanMismatchRecorded phase-2 replan, plan reopened
|
||||
projection survives the rotation
|
||||
18:01:02 planning session launches
|
||||
```
|
||||
|
||||
From that session's `.orchestra/launch.md`:
|
||||
|
||||
```text
|
||||
## Why this phase reopened
|
||||
|
||||
A plan was already accepted and the code contradicted it. Orchestra reopened
|
||||
this phase to settle that, and the session that found it is gone.
|
||||
|
||||
- phase: phase-2
|
||||
- observed: The summary lines are printed from the recorded results after
|
||||
every check has run, ...
|
||||
- the plan says: The plan states that phase 2 prints the header from a helper
|
||||
that returns the number of checks about to run.
|
||||
- evidence: scripts/orchestra_e2e_healthcheck.sh:1
|
||||
|
||||
The accepted plan stays accepted until you seal a replacement, and sealing one
|
||||
supersedes it along with every phase it had verified. Address the
|
||||
contradiction above: a replacement that repeats it will be contradicted again.
|
||||
```
|
||||
|
||||
It renders above the sealed research and plan, because it changes how they
|
||||
should be read. The first planning session of a task carries no such section,
|
||||
and neither does implement. Sealing the replacement (`2edcbe9e`) cleared the
|
||||
projection, which is the other end of its lifetime.
|
||||
|
||||
### The lesson worth keeping
|
||||
|
||||
Both rounds of this fix passed their unit tests. What separated them was where
|
||||
the assertion sat in the sequence. A projection written correctly and read one
|
||||
rotation later is not the same claim, and only the live run put the read where
|
||||
the agent does.
|
||||
|
||||
@@ -66,6 +66,14 @@ the live herdr instance and check.
|
||||
`docker logs orchestra-api`. **Deploying a code change means rebuilding the
|
||||
compose images** (`up -d --build`) — the running image can silently predate
|
||||
recent commits, so compare its build time against `git log`.
|
||||
- **A coordinator deploy does not deploy the console.** They are two images
|
||||
built from the same repo, and the usual `up -d --no-deps orchestra-api`
|
||||
leaves `orchestra-web-ui` on whatever it was. This has bitten twice: the
|
||||
ethos console landed in the repo on 2026-08-29 02:44 and was still serving a
|
||||
2026-07-30 image hours later, through two coordinator deploys. Rebuild it
|
||||
explicitly from the same clean worktree (`docker build` in `web/`, then
|
||||
`up -d --no-deps orchestra-web-ui`), and check the served bundle rather than
|
||||
the commit: `curl -s http://127.0.0.1:19145/assets/<css> | grep 8F7AE5`.
|
||||
- `orchestra.service` was the **previous** deployment; the unit file was
|
||||
deleted from `deploy/` on 2026-07-31 along with `redeploy.sh` (which
|
||||
`sudo install`ed to `/usr/local/bin` and restarted it). If a stale copy is
|
||||
|
||||
@@ -408,3 +408,113 @@ which is the priority function slice one deliberately omitted.
|
||||
|
||||
None of these required a schema commitment to discover. That was the point of
|
||||
making the first slice read-only.
|
||||
|
||||
## The invariant slice B was written against
|
||||
|
||||
**A projection must never manufacture provenance to make evidence easier to
|
||||
classify.** `task=None` is better than a confident lie.
|
||||
|
||||
Both defects the first live run of durable observations exposed were failures
|
||||
of exactly this rule, and both looked perfectly reasonable in code:
|
||||
|
||||
- **False attribution.** The worker's ring outlives the work it describes, so
|
||||
binding its entries to whatever task the worker is running now produced a
|
||||
clean, well-formed, wrong association between an old failure and an unrelated
|
||||
task. The task is read from the observation itself, and only an observation
|
||||
that names no task belongs to the current lease.
|
||||
- **Manufactured recurrence.** Treating "still present in the ring" as "it
|
||||
happened again" turned one failure into four incidents. Presence is not
|
||||
occurrence: an incident opens only when the entry advances past what has
|
||||
already been accounted for, and that high-water mark survives the close.
|
||||
|
||||
Either one would have corrupted the ledger while every number in it stayed
|
||||
plausible, which is the specific way this repo's bugs have always presented.
|
||||
|
||||
## Signatures are frozen at write time
|
||||
|
||||
The signature is computed when an incident is recorded and stored in the event.
|
||||
Re-projecting an old log with a newer normalizer therefore changes nothing:
|
||||
running one history through the build before and after the pane-name fix
|
||||
produced identical output.
|
||||
|
||||
That is the intended behavior. An event must not silently change meaning
|
||||
because normalization code changed. Repairing historical signatures is an
|
||||
explicit migration or reclassification event, never a different projection
|
||||
result from the same log.
|
||||
|
||||
## What counts as a manual intervention
|
||||
|
||||
An intervention is an operator action required to recover, repair, unblock or
|
||||
correct behaviour that should otherwise have proceeded autonomously. The
|
||||
ledger measures what the system costs to keep running, so routine operation
|
||||
does not belong in it.
|
||||
|
||||
Counts:
|
||||
|
||||
```text
|
||||
transaction_cleanup
|
||||
forced_release
|
||||
state_repair
|
||||
manual_requeue when recovery failed and a human had to requeue
|
||||
manual_phase_recovery
|
||||
worker_restart only when restarting is itself the repair
|
||||
```
|
||||
|
||||
Does not count:
|
||||
|
||||
```text
|
||||
deploy restart
|
||||
planned upgrade
|
||||
configuration rollout
|
||||
normal shutdown and start
|
||||
deliberate burn-in setup
|
||||
```
|
||||
|
||||
The distinction is policy rather than schema. `worker_restart` is the one kind
|
||||
that spans both sides, and it stays a single kind until someone actually
|
||||
misuses it; a field added before the confusion exists is a guess about how it
|
||||
will be misread.
|
||||
|
||||
The first live consequence: the deploy restart of `79d2053` was not recorded,
|
||||
and `manual_intervention` stayed in the gap list afterwards. That is the
|
||||
correct result. Orchestra can record the evidence and this history contains
|
||||
none, which is an honest gap rather than synthetic evidence.
|
||||
|
||||
## Slice B status, 2026-08-30
|
||||
|
||||
```text
|
||||
worker observation durability proven live
|
||||
incident open/close semantics proven live
|
||||
high-water, no phantom recurrence proven live
|
||||
task attribution proven live
|
||||
frozen historical provenance proven live
|
||||
|
||||
eligibility calculation proven against real data
|
||||
eligibility transition unexercised
|
||||
manual intervention recording implemented
|
||||
manual intervention provenance unexercised
|
||||
```
|
||||
|
||||
Neither unexercised branch will be manufactured. A second task hitting the
|
||||
same failure exercises the first, and a genuine repair exercises the second.
|
||||
|
||||
The proof that matters from the live run:
|
||||
|
||||
```text
|
||||
recurrence 5, intensity 10, distinct tasks 1, interventions 0 -> eligible=false
|
||||
```
|
||||
|
||||
A noisy failure inside one task does not become system-level maintenance debt.
|
||||
The three counts stay separate on purpose: recurrence is how often the incident
|
||||
happened, intensity is how repetitive each incident became, and breadth is how
|
||||
many independent tasks paid for it. Operational debt requires breadth unless an
|
||||
operator had to intervene.
|
||||
|
||||
### The reasons check is part of the design, not a formality
|
||||
|
||||
The aggregate numbers moved in a way that looked like a promotion, and reading
|
||||
them that way was wrong: the eligible count rose because of an unrelated item.
|
||||
The pure eligibility explanation contradicted that inference with counted
|
||||
facts. That is the argument for the ledger resting on mechanically derived
|
||||
evidence rather than on anyone's reading of what looks recurring, including an
|
||||
agent's.
|
||||
|
||||
@@ -0,0 +1,349 @@
|
||||
# Handoff: the release path is settled, and the UI became a truth detector
|
||||
|
||||
Written 2026-08-29, 14:20 local (10:20 UTC). Read with `BURNIN.md` (the run
|
||||
ledger, current through run 16), `DEBT-DESIGN.md`, `PLAN-SPEC-DESIGN.md`,
|
||||
`AUDIT.md` and `CLAUDE.md`. The previous handoff is
|
||||
`HANDOFF-2026-08-28-plan-v1.md`.
|
||||
|
||||
Everything below was observed live unless it says otherwise.
|
||||
|
||||
## The headline
|
||||
|
||||
**The expired-release defect is fixed and proven both ways.** The whole family
|
||||
around it is closed. F57 through F60 settle what happens to a release
|
||||
transaction in every case. That includes the ones that used to need an operator
|
||||
with a text editor.
|
||||
|
||||
**Three new things exist that did not before.** A bounded observation ring on
|
||||
worker health, closing F18. A read-only debt ledger projected from the event
|
||||
log. An operator console rebuilt on the ethos design system.
|
||||
|
||||
**The UI turned out to be a truth detector.** Nine screens were built against
|
||||
real endpoints. They found four places where Orchestra has no capability to
|
||||
support the intended interface. That list is the most valuable output of the
|
||||
session.
|
||||
|
||||
```text
|
||||
orchestra-plan-v1 plan machinery proven
|
||||
↓
|
||||
orchestra-f18-baseline d6ee10f, the bounded observation ring
|
||||
↓ 6 commits
|
||||
34f3c28 deployed now: release path settled, debt ledger, new UI
|
||||
```
|
||||
|
||||
## Deployed state
|
||||
|
||||
| Half | Revision |
|
||||
|---|---|
|
||||
| Coordinator, homesrv container | `34f3c28` |
|
||||
| Worker, workpc systemd | `34f3c28` |
|
||||
|
||||
```text
|
||||
commit 34f3c2888fc7d45d190d93e1ea42501b6cd3e474
|
||||
coordinator sha256 1d32d83ec859b36e12473fab01bbfc3c769b97b4f6ae4db2841db5161e0eda19
|
||||
worker sha256 0e3877321dea8a1eeda51ccc6f3ead1a14aa5f5cae4d95b704f61c364248ec65
|
||||
```
|
||||
|
||||
`cbd6b11` is one commit above and is documentation only. Three commits are
|
||||
unpushed.
|
||||
|
||||
**Worker installs no longer need a human.** The operator installed the
|
||||
`/etc/sudoers.d` line, so `sudo -n install …` and
|
||||
`sudo -n systemctl restart orchestra-worker` both work unattended. Verify the
|
||||
running revision from the journal, never the installed file.
|
||||
|
||||
## The defects fixed, and how each was found
|
||||
|
||||
Not one came from reading code. Every one came from a live run failing.
|
||||
|
||||
| Id | Commit | What |
|
||||
|---|---|---|
|
||||
| F57 | `6565b9f` | An expired lease could never commit the anchor it had already pushed. The worker sent an epoch the expiry replay had deleted, and the coordinator refused any handoff without a live lease. The epoch now belongs to the transaction, `TaskReleased` retains the ending epoch, and `lateHandoffAccepted` lets exactly that owner commit while the task is queued and unleased. |
|
||||
| F58 | `03663f4` | A superseded transaction retried a permanent 409 forever, holding the pane and pinning `ActiveTask`. Run 10's task did it for seven hours. `TaskLeased` now abandons a transaction whose id the lease does not carry. |
|
||||
| F59 | `8e37989` | F58 fires on `TaskLeased`, and a failed task is never leased again. `TaskFailed` now drops the transaction too. |
|
||||
| F60 | `3c7cf95` | The general rule the other two were reaching for. Terminal is failed or completed. Blocked keeps the transaction, because a reopen can still commit it, so `TaskBlocked` now retains the ending epoch as well. A refusal parks the commit for 30s backing off to 5 minutes, and any event about the task un-parks it. A transport failure is not an answer and retries at once. |
|
||||
| F18 | `d6ee10f` | The single `last_error` slot. Worker health now carries up to sixteen distinct observations with repeat counts and first/last times, collapsing by message rather than by position. |
|
||||
|
||||
### The rig that proved F57, and the guard
|
||||
|
||||
```text
|
||||
19:00:10.742 transaction opens at prepared, anchor pushing
|
||||
19:00:10.727 TaskReleased v15 reason=lease_expired surface=tui
|
||||
19:00:11.662 TaskReleased v16 the late commit, accepted 935ms after the lease died
|
||||
19:00:11.665 TaskLeased v17 successor picks up the handoff
|
||||
19:00:15.524 TaskPickupValidated v18
|
||||
```
|
||||
|
||||
Race guard, next boundary: force the expiry, then lease the task to a probe
|
||||
harness before the push finishes. The late commit is refused, no handoff is
|
||||
written, and the successor's lease stands.
|
||||
|
||||
**The rig technique matters more than the rig.** Suspending the worker cannot
|
||||
produce this ordering. The event replay runs at the top of every tick and
|
||||
discards the transaction. The ordering exists only inside one call:
|
||||
transaction opened, anchor pushing, commit not yet sent. So poll the worker
|
||||
state file at 2ms and fire `POST /v1/tasks/<id>/release` the instant a
|
||||
transaction appears at `prepared`.
|
||||
|
||||
**A named probe harness owns a lease without starting an agent.**
|
||||
`race-guard-probe` never picks anything up and expires on the normal TTL.
|
||||
|
||||
## Corrections to the previous handoff
|
||||
|
||||
**The operator lifecycle actions do not lose a version race.** On a leased
|
||||
task, `block`, `release` and `attention` are refused by
|
||||
`internal/store/store.go:885-901` when the payload omits `harness_id` and
|
||||
`lease_epoch`. Ten attempts in 550ms all failed that way. Send both fencing
|
||||
fields and they succeed on the first try.
|
||||
|
||||
**The OpenCode Zen free tier is not blocked.** The selected model was.
|
||||
|
||||
## The debt ledger
|
||||
|
||||
`DEBT-DESIGN.md` answers nine design questions and carries four amendments the
|
||||
operator made. Slice one is built, deployed and run.
|
||||
|
||||
**Slice one writes nothing.** A read-only projection over the existing log,
|
||||
plus a pure eligibility function and `GET /v1/debt`. It folded 881 events and
|
||||
produced 15 candidates and 3 gaps.
|
||||
|
||||
```text
|
||||
v1:operational:lease_expired:workpc-opencode:lease r=41 tasks=4
|
||||
v1:operational:lease_expired:workpc-claude:lease r=29 tasks=13
|
||||
v1:operational:lease_failure:-:lease r=20 tasks=16
|
||||
v1:correctness:handoff_validation:-:lease r=5 tasks=5
|
||||
```
|
||||
|
||||
The opencode failure shape is the top item, found mechanically. Run 14 reached
|
||||
the same conclusion by hand from a pane capture.
|
||||
|
||||
**It reported what it cannot see, which was the point.** The 409 release loop
|
||||
does not appear. That evidence lived in the F18 worker ring, and no event
|
||||
carries it. Manual interventions are a non-durable gap for the same reason.
|
||||
|
||||
**The first run exposed four defects in the model**, all recorded at the end of
|
||||
`DEBT-DESIGN.md`:
|
||||
|
||||
- the component part is too coarse for lease evidence
|
||||
- `harness` is often empty on block reasons
|
||||
- path normalization mangled a mismatch reference
|
||||
- recurrence alone is the wrong sort order
|
||||
|
||||
Layout, so the read model does not end up in the command layer:
|
||||
|
||||
```text
|
||||
internal/domain/debt.go types, signature, classification
|
||||
internal/store/debt_projection.go the fold, and the gap report
|
||||
internal/operations/debt.go CheckDebtEligibility
|
||||
```
|
||||
|
||||
## The operator console
|
||||
|
||||
Nine screens on the ethos system, signal violet `#8F7AE5`, routing fork motif.
|
||||
Each screen was built by its own agent against a foundation with one author.
|
||||
The shell, tokens and primitives could not drift into nine dialects.
|
||||
|
||||
**Render before signing off.** Three bugs existed that no computed value would
|
||||
have caught. All three came from looking at a screenshot:
|
||||
|
||||
- The previous stylesheet fought every shared class name and leaked properties
|
||||
the new rules never mention, which is how `position: fixed` survived on
|
||||
`.topbar`. It is now scoped under `.legacy` and reaches only the login route.
|
||||
That also stops its green accent and its `backdrop-filter` from reaching the
|
||||
console.
|
||||
- Go marshals a zero `time.Time` as `0001-01-01T00:00:00Z` and `omitempty` does
|
||||
not omit a struct, so absent timestamps arrived populated-looking and
|
||||
rendered as `739855d ago`. Stripped once in `client.ts`, with a test.
|
||||
- Long machine ids overflowed their cards and painted under the next one.
|
||||
|
||||
Chromium is installed at `/usr/bin/chromium`. To see a screen without a live
|
||||
session, write a throwaway harness that stubs `window.fetch` and renders
|
||||
`<Console>` inside a `MemoryRouter`, served by vite on a spare port. Note that
|
||||
`npx` and `./node_modules/.bin/*` do not work on this filesystem: call
|
||||
`node ./node_modules/vite/bin/vite.js` directly.
|
||||
|
||||
## What the UI proved Orchestra cannot do
|
||||
|
||||
This is the part worth acting on. Each screen refused to fake something, and
|
||||
the refusals name real capability gaps.
|
||||
|
||||
| Gap | Evidence |
|
||||
|---|---|
|
||||
| **No web-facing human-decision write path** | `Steer / Correct` is disabled. `internal/ui/ui.go`'s action switch has grant/deny approval, resubmit, handoff, release, block and complete, and nothing writes `HumanDecisionRecorded`. The spec makes steering the primary action of the task detail screen. |
|
||||
| **No keystroke forwarding** | `Take control` is disabled. Only resubmit and approval grant/deny reach a live pane. |
|
||||
| **Context occupancy is trapped in herdr** | Three screens independently hit it. No projection carries it. |
|
||||
| **Project configuration is not served** | Repo, remote, quality gate and verification policy live only in `config.jsonc`. The projects screen can show none of it. |
|
||||
| **The web cannot request a handoff for a federated task** | `RequestHandoff` needs a local coordinator and answers 503. That is the Design A guardrail working. |
|
||||
|
||||
The operator's direction on these. Treat first-class direct human input as the
|
||||
highest-value backend feature. Build it as `POST /v1/tasks/<id>/decisions`,
|
||||
using the same durable decision semantics as Gitea comments, so Gitea, CLI and
|
||||
web converge on one `HumanDecisionRecorded`. Keep take-control disabled,
|
||||
because arbitrary pane input bypasses the durable authority model. Expose
|
||||
occupancy through a session health projection rather than teaching the web
|
||||
server about herdr. Add a read-only effective project configuration endpoint,
|
||||
which F61 will also need.
|
||||
|
||||
## Where the plan-machinery ladder stands
|
||||
|
||||
Proven in run 14: the worker executes the **sealed plan's** commands rather
|
||||
than the request's, and every `PlanPhaseVerified` binds `plan_ref`, `phase_id`,
|
||||
`at_sha`, `evidence_ref`, `lease_epoch` and `harness_id`.
|
||||
|
||||
Proven in run 16: a successor inherits the **whole sealed plan**, all phases
|
||||
with their files, changes, verification, and the research citations.
|
||||
|
||||
**Still unproven, and the next runtime item:**
|
||||
|
||||
```text
|
||||
mid-implement rotation
|
||||
→ successor picks up in implement
|
||||
→ launch context states which phases are already verified
|
||||
|
||||
manual verification
|
||||
→ SHA goes stale
|
||||
→ re-verification
|
||||
|
||||
plan mismatch
|
||||
→ human decision
|
||||
→ real replan, old plan retained, replacement launched
|
||||
```
|
||||
|
||||
Two things make the first one hard, and both are now known:
|
||||
|
||||
- **A trivial task spends about four minutes in implement**, and verifies every
|
||||
phase against one tree near the end. The implementer writes the whole change
|
||||
first, then verifies each phase in turn. Use a task whose implement phase
|
||||
genuinely runs long.
|
||||
- **A state-file edit does not survive a running worker.** It holds sessions in
|
||||
memory and writes them back. Suspend, edit, then `sudo systemctl restart`.
|
||||
Resuming lets the old copy win. Setting `handoff_requested` on the session is
|
||||
the production rotation lever.
|
||||
|
||||
Progress is withheld from a **review** successor on purpose
|
||||
(`internal/agentctx/agentctx.go:321`), because an independent review must
|
||||
reconstruct the change from the diff. That absence is not the defect.
|
||||
|
||||
## F61 and F62, recorded and not built
|
||||
|
||||
**F61: the planner learns the verification policy by refusal.** The brief says
|
||||
a policy exists, not what is in it. Every plan therefore pays one refused round
|
||||
trip. The recovery loop works: run 14's planner consumed the refusal and
|
||||
resealed 31 seconds later. It is not a one-liner, because `agentctx.Input.Policy` is filled
|
||||
from the worker's `SafeOperations` while the verification policy is
|
||||
coordinator-side. The reason to promote it later is local models, which may
|
||||
propose forbidden commands repeatedly because they cannot infer the allowed
|
||||
substitute.
|
||||
|
||||
**F62: a requested handoff nobody answers is invisible.** The rotation is
|
||||
agent-driven. In run 16 the agent never wrote `HANDOFF.md`, renewals stopped,
|
||||
the lease expired, and the task lost an attempt. Worker health recorded only
|
||||
`agent status idle and pane unchanged`, 34 times. There is no timeout, no
|
||||
retry, and no observation saying a handoff was requested and left unanswered.
|
||||
The expiry is indistinguishable from an ordinary idle one.
|
||||
|
||||
## Live state
|
||||
|
||||
Three pull requests are open and unreviewed:
|
||||
|
||||
```text
|
||||
06G4M6HF1Z3EREX1X3NEKSHP24 pulls/18
|
||||
06G4M8WHGQ4P3GQMPEEH0RJRHM pulls/19
|
||||
06G4SWEVP71FYYKAV5FV0ZK5ZG pulls/20
|
||||
```
|
||||
|
||||
Both workers are online on `34f3c28` with no release transactions and no
|
||||
sessions. 29 blocked `test-e2e` tasks are burn-in debris. Three `correx` tasks
|
||||
are queued and unschedulable, because `correx` has no entry in the
|
||||
coordinator's `config.jsonc`.
|
||||
|
||||
**opencode now runs.** The model was the problem, and `hy3-free` works. It then
|
||||
stops on a permission prompt, because `~/.config/opencode/opencode.jsonc` sets
|
||||
`"bash": "ask"`. Orchestra can answer that exact dialog, but only when an
|
||||
operator queues `grant_approval`, so unattended work stalls on the first
|
||||
command. Set `"bash": "allow"` for unattended runs. That file also has no
|
||||
top-level `model` key, so OpenCode picks whatever sits at the top of
|
||||
`~/.local/state/opencode/model.json`, which any manual pick silently changes.
|
||||
|
||||
**The opencode adapter is now a debt item, not a curiosity.** It cannot resolve
|
||||
a session file. Activity therefore reads `unknown`, and the worker cannot tell
|
||||
finished from never-started. The debt projection surfaced it independently as
|
||||
the top recurring operational item.
|
||||
|
||||
## Things that will bite
|
||||
|
||||
- **Background python tasks get killed here.** Three watchers died before doing
|
||||
anything. Foreground polling and the `Monitor` tool both work.
|
||||
- **`/v1/events` is one line of JSON.** A `grep` for two substrings matches
|
||||
across unrelated tasks. Parse it.
|
||||
- **`npx` and `./node_modules/.bin/*` fail on this filesystem.** Call node
|
||||
directly.
|
||||
- **`rm` and `cp` are interactive.** Use `/bin/rm -f` and `install`.
|
||||
- **Secrets are guarded.** Expand a token inside the container in one remote
|
||||
command:
|
||||
`T=$(docker exec orchestra-api printenv ORCHESTRA_TUI_TOKEN); curl -s -H "X-Orchestra-Surface: tui" -H "Authorization: Bearer $T" ...`
|
||||
- **Rebuild the coordinator from a detached worktree**, and split the worktree
|
||||
add, the docker build and the compose up into separate commands.
|
||||
- **`deploy/build.sh` builds both halves from one commit with one stamp.** Use
|
||||
it.
|
||||
- **Geist is not on disk.** Both stacks fall back to system faces, and the
|
||||
ethos threat model rules out the font CDN.
|
||||
|
||||
## Resume in this order
|
||||
|
||||
Set by the operator at the session boundary. Do not rebuild it from the commits.
|
||||
|
||||
1. **F62 first.** Make a requested-but-unanswered handoff visible and bounded.
|
||||
Preserve the lease while Orchestra is explicitly waiting, retry the confirmed
|
||||
request, then emit a causal timeout. Today it ends as generic idleness.
|
||||
2. **Repeat the implement-successor rung** with a deliberately longer task. Get
|
||||
at least one `PlanPhaseVerified`, force the handoff while still in
|
||||
`implement`, and prove the successor sees the complete accepted plan, the
|
||||
verified previous phase, the first unfinished phase, and the current human
|
||||
authority.
|
||||
3. **Finish the remaining ladder.** Manual verification, SHA staleness and
|
||||
reverification, human-decision mismatch, then a real replan with old-plan
|
||||
provenance.
|
||||
4. **Use the capability table above as backend work discovery.** Do not add fake
|
||||
controls. Each disabled action is concrete evidence of a missing capability.
|
||||
5. **Continue the debt slices independently.** Durable worker observations and
|
||||
manual-intervention events are the next evidence gaps. Not automatic
|
||||
maintenance yet.
|
||||
|
||||
## What the tag means
|
||||
|
||||
```text
|
||||
orchestra-release-v1 -> 34f3c28
|
||||
deployed, settled lifecycle plus the truth-detector UI baseline
|
||||
|
||||
later HEADs
|
||||
experimental plan, debt and runtime work that must earn their own
|
||||
release proof
|
||||
```
|
||||
|
||||
Use that distinction. A defect found in experimental work is not a reason to
|
||||
reopen settled architecture.
|
||||
|
||||
## The roadmap the operator set
|
||||
|
||||
```text
|
||||
A. runtime correctness finish the plan-machinery live proof
|
||||
B. evidence and debt durable worker observations, manual-intervention
|
||||
events, then rerun the projection
|
||||
C. operator surface first-class HumanDecision write API,
|
||||
effective project-config read API,
|
||||
session and context health projection
|
||||
D. adapter fix opencode activity and session resolution
|
||||
E. UI wire capabilities as backend support becomes real
|
||||
```
|
||||
|
||||
The framing that ties them together is the thing to keep. The UI says what
|
||||
Orchestra cannot expose or control. The debt ledger says which of those
|
||||
shortcomings repeatedly costs something. The burn-in says which runtime
|
||||
semantics are reliable. Those three decide what gets built next.
|
||||
|
||||
One design question to settle before B's slice two writes any code. The worker
|
||||
ring is bounded and lossy by construction. Ingesting it durably means deciding
|
||||
whether the coordinator stores every observation as an event, or only
|
||||
transitions. Storing every heartbeat's ring would write the same 41-count
|
||||
observation hundreds of times.
|
||||
@@ -0,0 +1,17 @@
|
||||
# orchestra ui mockups
|
||||
|
||||
`final/` contains the nine accepted/current screen directions:
|
||||
|
||||
1. dashboard
|
||||
2. task detail
|
||||
3. terminal live pane
|
||||
4. tasks
|
||||
5. decisions
|
||||
6. workers
|
||||
7. projects
|
||||
8. review
|
||||
9. settings
|
||||
|
||||
`iterations/` contains every generated mockup from the design session, including superseded variants.
|
||||
|
||||
`orchestra-ui-spec.md` is the accompanying implementation/design specification.
|
||||
@@ -0,0 +1,56 @@
|
||||
// Runs the debt projection over a captured event log and prints one row per
|
||||
// item, so two builds can be compared on identical input. The point of the
|
||||
// comparison is the signature fix: cross-task incidents must collapse into one
|
||||
// item while genuinely different failures stay separate.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
|
||||
"orchestra/internal/domain"
|
||||
"orchestra/internal/operations"
|
||||
"orchestra/internal/store"
|
||||
)
|
||||
|
||||
func main() {
|
||||
b, err := os.ReadFile(os.Args[1])
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
var events []domain.Event
|
||||
if err := json.Unmarshal(b, &events); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
ledger := store.ProjectDebt(events)
|
||||
eligible := map[string]operations.DebtCandidate{}
|
||||
for _, c := range operations.EligibleDebt(ledger) {
|
||||
eligible[c.Item.ID] = c
|
||||
}
|
||||
rows := make([]map[string]any, 0, len(ledger.Items))
|
||||
for _, item := range ledger.Items {
|
||||
tasks := map[string]bool{}
|
||||
intensity := 0
|
||||
for _, o := range item.Observations {
|
||||
if o.TaskID != "" {
|
||||
tasks[o.TaskID] = true
|
||||
}
|
||||
intensity += o.Repeats
|
||||
}
|
||||
row := map[string]any{
|
||||
"signature": item.ID, "class": string(item.Class),
|
||||
"recurrence": len(item.Observations), "intensity": intensity,
|
||||
"tasks": len(tasks), "eligible": false, "reasons": []string{},
|
||||
}
|
||||
check := operations.CheckDebtEligibility(item)
|
||||
row["eligible"] = check.Eligible
|
||||
row["reasons"] = check.Reasons
|
||||
_ = eligible
|
||||
rows = append(rows, row)
|
||||
}
|
||||
sort.Slice(rows, func(i, j int) bool { return rows[i]["signature"].(string) < rows[j]["signature"].(string) })
|
||||
out, _ := json.MarshalIndent(map[string]any{"items": rows, "gaps": ledger.Gaps}, "", " ")
|
||||
fmt.Println(string(out))
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"orchestra/internal/domain"
|
||||
"orchestra/internal/federation"
|
||||
"orchestra/internal/herdr"
|
||||
)
|
||||
|
||||
// F62. Run 16: the agent was asked to hand off, never wrote HANDOFF.md,
|
||||
// renewals stopped, and the lease died as ordinary idleness. Waiting is now
|
||||
// bounded: re-ask once, then give the task up with a class that says why.
|
||||
func TestUnansweredHandoffIsRetriedThenGivenUp(t *testing.T) {
|
||||
var nack map[string]any
|
||||
w, backend, _, done := phaseWorker(t, func(rw http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/nack") {
|
||||
_ = json.NewDecoder(r.Body).Decode(&nack)
|
||||
}
|
||||
rw.Write([]byte(`{}`))
|
||||
})
|
||||
defer done()
|
||||
ctx := context.Background()
|
||||
requested := func(ago time.Duration) herdr.Session {
|
||||
s := w.sessions["task"]
|
||||
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
|
||||
s.HandoffRequestedAt = time.Now().UTC().Add(-ago)
|
||||
w.sessions["task"] = s
|
||||
return s
|
||||
}
|
||||
|
||||
// Still inside the answering window: nothing said, nothing given up.
|
||||
if s, gaveUp := w.watchHandoff(ctx, "task", requested(time.Minute)); gaveUp || s.HandoffRetried {
|
||||
t.Fatalf("gave up while still waiting: gaveUp=%v session=%+v", gaveUp, s)
|
||||
}
|
||||
if len(backend.prompts) != 0 {
|
||||
t.Fatalf("re-asked too early: %q", backend.prompts)
|
||||
}
|
||||
|
||||
// Past the retry point: asked again, exactly once.
|
||||
s, gaveUp := w.watchHandoff(ctx, "task", requested(handoffRetryAfter+time.Minute))
|
||||
if gaveUp || !s.HandoffRetried || len(backend.prompts) != 1 {
|
||||
t.Fatalf("retry: gaveUp=%v retried=%v prompts=%q", gaveUp, s.HandoffRetried, backend.prompts)
|
||||
}
|
||||
if _, gaveUp = w.watchHandoff(ctx, "task", s); gaveUp || len(backend.prompts) != 1 {
|
||||
t.Fatalf("re-asked every tick: %q", backend.prompts)
|
||||
}
|
||||
|
||||
// Past the bound: a causal reclaim, and no lease left to renew.
|
||||
s = requested(handoffAnswerTimeout + time.Second)
|
||||
s.HandoffRetried = true
|
||||
w.sessions["task"] = s
|
||||
if _, gaveUp = w.watchHandoff(ctx, "task", s); !gaveUp {
|
||||
t.Fatal("an unanswered handoff waited forever")
|
||||
}
|
||||
if nack["failure_class"] != "handoff_unanswered" {
|
||||
t.Fatalf("nack = %+v", nack)
|
||||
}
|
||||
if detail, _ := nack["last_error"].(string); !strings.Contains(detail, "phase_changed") {
|
||||
t.Fatalf("the reclaim does not name the request: %q", detail)
|
||||
}
|
||||
if _, held := w.leases["task"]; held {
|
||||
t.Fatal("the given-up task kept its lease")
|
||||
}
|
||||
}
|
||||
|
||||
// The lease must survive the wait it was asked to make: an idle pane is the
|
||||
// answer Orchestra requested, not evidence of an agent that stopped working.
|
||||
func TestWaitingForAHandoffKeepsTheLease(t *testing.T) {
|
||||
renewals := 0
|
||||
api := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
|
||||
renewals++
|
||||
rw.Write([]byte(`{}`))
|
||||
}))
|
||||
defer api.Close()
|
||||
backend := &recordingBackend{status: "idle", progress: "same screen"}
|
||||
w := &worker{
|
||||
api: federation.Client{BaseURL: api.URL, WorkerID: "h", Token: "t"},
|
||||
backend: backend,
|
||||
harness: "claude",
|
||||
sessions: map[string]herdr.Session{"task": {PaneID: "pane", HandoffRequested: true, HandoffRequestedAt: time.Now().UTC()}},
|
||||
leases: map[string]lease{"task": {Epoch: "e", Version: 1, Until: time.Now(), ProgressSHA: domain.Hash([]byte("same screen"))}},
|
||||
quarantined: map[string]bool{},
|
||||
statePath: filepath.Join(t.TempDir(), "state.json"),
|
||||
}
|
||||
w.renewLeases(context.Background())
|
||||
if renewals != 1 {
|
||||
t.Fatalf("a lease waiting on a requested handoff renewed %d times, want 1", renewals)
|
||||
}
|
||||
|
||||
// Past the bound the exemption stops: watchHandoff has given the task up
|
||||
// by then, and nothing keeps an unanswered request alive.
|
||||
s := w.sessions["task"]
|
||||
s.HandoffRequestedAt = time.Now().UTC().Add(-handoffAnswerTimeout - time.Second)
|
||||
w.sessions["task"] = s
|
||||
l := w.leases["task"]
|
||||
l.Until = time.Now()
|
||||
w.leases["task"] = l
|
||||
w.renewLeases(context.Background())
|
||||
if renewals != 1 {
|
||||
t.Fatalf("the exemption outlived its bound: renewals=%d", renewals)
|
||||
}
|
||||
}
|
||||
@@ -103,8 +103,13 @@ func (w *worker) recordError(err error) {
|
||||
w.observations = append(w.observations, federation.Observation{Message: msg, Count: 1, First: now, Last: now})
|
||||
}
|
||||
|
||||
// workerIncarnation identifies this process. A restarted worker cannot
|
||||
// continue the previous process's failures, and nothing else on the wire says
|
||||
// a restart happened: build revision and worker id both survive it.
|
||||
var workerIncarnation = domain.NewID()
|
||||
|
||||
func (w *worker) health(ctx context.Context) federation.WorkerHealth {
|
||||
h := federation.WorkerHealth{HerdrStatus: "unknown"}
|
||||
h := federation.WorkerHealth{HerdrStatus: "unknown", Incarnation: workerIncarnation}
|
||||
if backend := w.executionBackend(); backend != nil {
|
||||
h.Backend = backend.Kind()
|
||||
}
|
||||
@@ -174,6 +179,7 @@ func releaseBackoff(attempts int) time.Duration {
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
type projectConfig struct {
|
||||
Repo string `json:"repo"`
|
||||
Root string `json:"worktree_root"`
|
||||
@@ -673,6 +679,13 @@ func (w *worker) releaseReady(ctx context.Context) {
|
||||
w.advanceRelease(ctx, id, s)
|
||||
continue
|
||||
}
|
||||
if s.HandoffRequested {
|
||||
next, gaveUp := w.watchHandoff(ctx, id, s)
|
||||
if gaveUp {
|
||||
continue
|
||||
}
|
||||
s = next
|
||||
}
|
||||
w.rotationTick(ctx, id, s)
|
||||
}
|
||||
}
|
||||
@@ -772,7 +785,7 @@ func (w *worker) rotationTick(ctx context.Context, id string, s herdr.Session) {
|
||||
w.recordError(fmt.Errorf("rotation %s threshold prompt: %w", id, err))
|
||||
return
|
||||
}
|
||||
s.HandoffRequested, s.HandoffReason = true, d.Reason
|
||||
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, d.Reason, time.Now().UTC()
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
}
|
||||
@@ -1166,6 +1179,70 @@ func (w *worker) submit(ctx context.Context, id string, s herdr.Session, e compl
|
||||
// the directory also holds for a worktree that has no inner .gitignore.
|
||||
const stageExclude = ":!.orchestra"
|
||||
|
||||
// F62: a requested handoff nobody answers was invisible. Renewals stopped,
|
||||
// the lease expired, and the task lost an attempt with nothing on record
|
||||
// saying a handoff had ever been asked for — worker health showed only "agent
|
||||
// status idle and pane unchanged", 34 times in run 16.
|
||||
const (
|
||||
handoffRetryAfter = 4 * time.Minute
|
||||
handoffAnswerTimeout = 10 * time.Minute
|
||||
)
|
||||
|
||||
// watchHandoff bounds the wait for an agent's handoff answer: re-send the
|
||||
// request once, then give the task up with a class that names the cause. It
|
||||
// returns the session to keep using and whether the task was given up.
|
||||
func (w *worker) watchHandoff(ctx context.Context, id string, s herdr.Session) (herdr.Session, bool) {
|
||||
if s.HandoffRequestedAt.IsZero() {
|
||||
// A session persisted before the stamp existed, or requested by a path
|
||||
// that does not set it. Start the clock now rather than time out a
|
||||
// request retroactively.
|
||||
s.HandoffRequestedAt = time.Now().UTC()
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
return s, false
|
||||
}
|
||||
waited := time.Since(s.HandoffRequestedAt)
|
||||
if waited < handoffRetryAfter {
|
||||
return s, false
|
||||
}
|
||||
if waited < handoffAnswerTimeout {
|
||||
if s.HandoffRetried || w.executionBackend() == nil {
|
||||
return s, false
|
||||
}
|
||||
a := herdr.CLIAdapter{Backend: w.executionBackend(), Harness: w.harness}
|
||||
var err error
|
||||
if s.HandoffReason != "" {
|
||||
err = a.RequestHandoffReason(ctx, s, s.HandoffReason, nil)
|
||||
} else {
|
||||
err = a.RequestHandoff(ctx, s)
|
||||
}
|
||||
if err != nil {
|
||||
w.recordError(fmt.Errorf("handoff %s re-request: %w", id, err))
|
||||
return s, false
|
||||
}
|
||||
s.HandoffRetried = true
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
w.recordError(fmt.Errorf("handoff %s (%s) unanswered for %s: request re-sent", id, s.HandoffReason, waited.Round(time.Second)))
|
||||
return s, false
|
||||
}
|
||||
l, ok := w.leases[id]
|
||||
if !ok {
|
||||
return s, false
|
||||
}
|
||||
detail := fmt.Sprintf("handoff requested (%s) and unanswered for %s", s.HandoffReason, waited.Round(time.Second))
|
||||
if err := w.api.NackStart(ctx, id, l.Epoch, l.Version, "handoff_unanswered", detail, w.sessionEvidence(ctx, id, s)); err != nil {
|
||||
w.recordError(fmt.Errorf("handoff timeout %s: %w", id, err))
|
||||
return s, false
|
||||
}
|
||||
// The coordinator answers with TaskReleased; its replay quarantines the
|
||||
// pane. Drop the lease here so nothing renews it in the meantime.
|
||||
delete(w.leases, id)
|
||||
_ = w.save()
|
||||
w.recordError(errors.New(detail))
|
||||
return s, true
|
||||
}
|
||||
|
||||
func (w *worker) renewLeases(ctx context.Context) {
|
||||
if w.executionBackend() == nil {
|
||||
return
|
||||
@@ -1199,6 +1276,11 @@ func (w *worker) renewLeases(ctx context.Context) {
|
||||
case l.ProgressSHA == "":
|
||||
// First renewal has no baseline to compare against. Record one and
|
||||
// allow this renewal; the next one must show real movement.
|
||||
case s.HandoffRequested && !s.HandoffRequestedAt.IsZero() && time.Since(s.HandoffRequestedAt) < handoffAnswerTimeout:
|
||||
// Orchestra told this agent to stop and write its handoff. A quiet
|
||||
// pane is the answer it was asked for, so the lease is held while
|
||||
// the wait is explicitly bounded (F62). Past the bound the case
|
||||
// stops matching and watchHandoff has already given the task up.
|
||||
default:
|
||||
w.recordError(fmt.Errorf("lease %s not renewed: agent status %s and pane unchanged since the last renewal", taskID, status))
|
||||
continue
|
||||
@@ -1807,7 +1889,7 @@ func main() {
|
||||
window: window,
|
||||
// Capacity stays one per identity because a herdr's declared
|
||||
// concurrency is one. Serving N harnesses gives the process N slots.
|
||||
registration: federation.Worker{ID: spec.ID, Address: spec.Address, Capacity: 1, SupportedProjects: supported, Build: buildinfo.Current()},
|
||||
registration: federation.Worker{ID: spec.ID, Address: spec.Address, Capacity: 1, SupportedProjects: supported, Build: buildinfo.Current(), Incarnation: workerIncarnation},
|
||||
}
|
||||
if w.registration.Address == "" {
|
||||
w.registration.Address = os.Getenv("ORCHESTRA_WORKER_ADDRESS")
|
||||
@@ -1932,7 +2014,7 @@ func (w *worker) federatedTurn(ctx context.Context, id string, a herdr.Adapter,
|
||||
w.recordError(fmt.Errorf("reconcile failure handoff %s: %w", id, err))
|
||||
return
|
||||
}
|
||||
s.HandoffRequested, s.HandoffReason = true, "reconcile_failure"
|
||||
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "reconcile_failure", time.Now().UTC()
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
return
|
||||
@@ -2009,9 +2091,18 @@ func (w *worker) rotateForPhase(ctx context.Context, id string, a herdr.Adapter,
|
||||
w.recordError(fmt.Errorf("phase rotation %s: %w", id, err))
|
||||
return
|
||||
}
|
||||
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
|
||||
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "phase_changed", time.Now().UTC()
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
// A request belongs to the session that wrote it. Both files sit in the
|
||||
// worktree, which outlives the session, so a successor in a different
|
||||
// phase would find and execute them: that is how a reopened planning
|
||||
// session verified a phase of the plan it was replacing.
|
||||
for _, name := range []string{planProgressFile, phaseRequestFile} {
|
||||
if err := os.Remove(filepath.Join(s.Worktree, ".orchestra", name)); err != nil && !os.IsNotExist(err) {
|
||||
w.recordError(fmt.Errorf("phase rotation %s: drop %s: %w", id, name, err))
|
||||
}
|
||||
}
|
||||
log.Printf("phase changed for %s: session rotating", id)
|
||||
}
|
||||
|
||||
|
||||
@@ -457,3 +457,32 @@ func TestPlanVerificationRunsThePlansCommandsAndReportsExitCodes(t *testing.T) {
|
||||
t.Fatalf("the outcome was not delivered: %v", backend.prompts)
|
||||
}
|
||||
}
|
||||
|
||||
// A request belongs to the session that wrote it. The worktree outlives the
|
||||
// session, so a rotation that leaves these files behind hands them to a
|
||||
// successor running in a different phase (run 20).
|
||||
func TestRotationDropsTheEndedSessionsRequests(t *testing.T) {
|
||||
w, backend, wt, done := phaseWorker(t, func(rw http.ResponseWriter, r *http.Request) {
|
||||
rw.Write([]byte(`{}`))
|
||||
})
|
||||
defer done()
|
||||
progress := filepath.Join(wt, ".orchestra", planProgressFile)
|
||||
request := filepath.Join(wt, ".orchestra", phaseRequestFile)
|
||||
for _, p := range []string{progress, request} {
|
||||
if err := os.WriteFile(p, []byte(`{"phase":"phase-1","status":"ready_for_verification"}`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
a := herdr.CLIAdapter{Backend: backend, Harness: "claude"}
|
||||
w.rotateForPhase(context.Background(), "task", a, w.sessions["task"])
|
||||
|
||||
for _, p := range []string{progress, request} {
|
||||
if _, err := os.Stat(p); !os.IsNotExist(err) {
|
||||
t.Fatalf("%s survived the rotation that ended the session that wrote it", filepath.Base(p))
|
||||
}
|
||||
}
|
||||
if s := w.sessions["task"]; !s.HandoffRequested || s.HandoffReason != "phase_changed" {
|
||||
t.Fatalf("the session was not rotated: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -228,6 +228,9 @@ func main() {
|
||||
// Pull-request readers by source name, for reflecting submitted work.
|
||||
pullRequests := map[string]human.PullRequestSource{}
|
||||
localMachine := os.Getenv("ORCHESTRA_MACHINE_ID")
|
||||
// One tracker for the process: it holds the per-incident accumulation that
|
||||
// makes an evicted and recreated ring entry add up instead of restarting.
|
||||
observations := &operations.ObservationTracker{Store: s}
|
||||
workers := &federation.Registry{AdmitToken: os.Getenv("ORCHESTRA_FEDERATION_ADMIT_TOKEN"), StatePath: filepath.Join(dir, "federation-state.json")}
|
||||
if err := workers.Load(); err != nil {
|
||||
log.Fatalf("load federation state: %v", err)
|
||||
@@ -445,6 +448,49 @@ func main() {
|
||||
}
|
||||
json.NewEncoder(w).Encode(s.Events(n))
|
||||
})
|
||||
mux.HandleFunc("/v1/interventions", func(w http.ResponseWriter, r *http.Request) {
|
||||
// The one command an operator has for saying "I fixed this by hand".
|
||||
// Everything Orchestra does to itself is already an event; a manual
|
||||
// repair is the only kind of recovery that leaves no trace unless the
|
||||
// person who made it says so.
|
||||
if r.Method == http.MethodGet {
|
||||
out := make([]domain.Event, 0)
|
||||
for _, e := range s.Events(0) {
|
||||
if e.Type == domain.EventOperatorInterventionRecorded {
|
||||
out = append(out, e)
|
||||
}
|
||||
}
|
||||
json.NewEncoder(w).Encode(out)
|
||||
return
|
||||
}
|
||||
if r.Method != http.MethodPost {
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
if err := authz.AuthorizeEvent(surface(r), domain.EventOperatorInterventionRecorded); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
var in domain.OperatorIntervention
|
||||
if json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<16)).Decode(&in) != nil {
|
||||
http.Error(w, "invalid json", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
e, err := operations.RecordIntervention(s, surface(r), in)
|
||||
switch {
|
||||
case errors.Is(err, domain.ErrNotFound):
|
||||
http.Error(w, err.Error(), http.StatusNotFound)
|
||||
return
|
||||
case errors.Is(err, domain.ErrInvalid):
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
case err != nil:
|
||||
http.Error(w, err.Error(), http.StatusConflict)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
json.NewEncoder(w).Encode(e)
|
||||
})
|
||||
mux.HandleFunc("/v1/debt", func(w http.ResponseWriter, r *http.Request) {
|
||||
// Read-only, and deliberately so. This projection is evidence about
|
||||
// history, not a new kind of truth: nothing here writes an event,
|
||||
@@ -1391,6 +1437,29 @@ func main() {
|
||||
http.Error(w, err.Error(), 404)
|
||||
return
|
||||
}
|
||||
// The ring carries messages and nothing else, so the coordinator
|
||||
// attributes them: the worker's active task, and that task's
|
||||
// current lease epoch, are what bind an incident to the work it
|
||||
// happened during. A failed fold is logged rather than failing the
|
||||
// heartbeat, because losing evidence about a worker is not a reason
|
||||
// to also stop believing the worker is alive.
|
||||
report := operations.WorkerReport{
|
||||
WorkerID: parts[3], Incarnation: health.Incarnation,
|
||||
TaskID: health.ActiveTask, At: time.Now().UTC(),
|
||||
}
|
||||
if report.TaskID != "" {
|
||||
if t, ok := s.Task(report.TaskID); ok && t.Lease != nil {
|
||||
report.LeaseEpoch = t.Lease.Epoch
|
||||
}
|
||||
}
|
||||
for _, o := range health.Observations {
|
||||
report.Observations = append(report.Observations, domain.WorkerObservation{
|
||||
Message: o.Message, Count: o.Count, First: o.First, Last: o.Last,
|
||||
})
|
||||
}
|
||||
if _, err := observations.Ingest(report); err != nil {
|
||||
log.Printf("observation incidents for %s: %v", parts[3], err)
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
return
|
||||
}
|
||||
@@ -1589,6 +1658,12 @@ func main() {
|
||||
case "invalid_handoff":
|
||||
typ = "TaskBlocked"
|
||||
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonHandoffValidation), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_nacked", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
||||
case "handoff_unanswered":
|
||||
// F62: not a launch failure. The agent was asked to hand off
|
||||
// and never did, so the reclaim says exactly that instead of
|
||||
// arriving as an ordinary idle expiry.
|
||||
typ = "TaskReleased"
|
||||
p, _ = json.Marshal(map[string]any{"reason": "handoff_unanswered", "failure_class": b.FailureClass, "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "handoff_unanswered", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
||||
case "launch_uncertain":
|
||||
typ = "TaskNeedsAttention"
|
||||
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonLeaseFailure), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_uncertain", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
||||
@@ -1825,6 +1900,25 @@ func main() {
|
||||
coordinator.ReconcileFailureHandoff = v
|
||||
}
|
||||
}
|
||||
// The third reconciliation point, and the same reasoning the submitted
|
||||
// work loop above already uses: a blocked task cannot be leased, so a
|
||||
// pre-lease hook can never see the reply that should unblock it (F64).
|
||||
// Orchestra stopped and asked the human a question; nothing else here
|
||||
// will go and read the answer.
|
||||
go func() {
|
||||
ticker := time.NewTicker(time.Minute)
|
||||
defer ticker.Stop()
|
||||
for range ticker.C {
|
||||
for _, t := range s.Tasks() {
|
||||
if t.State != domain.StateBlocked || !domain.BlockReasonAwaitsReply(t.BlockReason) {
|
||||
continue
|
||||
}
|
||||
if err := reconciler.Reconcile(context.Background(), t.ID); err != nil {
|
||||
log.Printf("reconcile blocked task %s: %v", t.ID, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
if path := os.Getenv("ORCHESTRA_JSONL"); path != "" {
|
||||
sup := &provider.Supervisor{Name: "jsonl", Run: func(ctx context.Context) error {
|
||||
|
||||
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
@@ -314,6 +314,9 @@ func renderTask(in Input) string {
|
||||
fmt.Fprintf(&b, "- head: %s\n", fallback(in.Git.HeadSHA))
|
||||
fmt.Fprintf(&b, "- uncommitted changes: %t\n", in.Git.Dirty)
|
||||
|
||||
// Above the sealed artifacts, because it is the reason this session exists
|
||||
// and it changes how the artifacts below should be read.
|
||||
b.WriteString(renderReopen(in))
|
||||
b.WriteString(renderSealed(in))
|
||||
// Below the plan, above continuity. Progress is a fact about the plan, so
|
||||
// it follows the plan; continuity is one predecessor's account, so it
|
||||
@@ -499,6 +502,35 @@ func renderPlanProgress(in Input) string {
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// renderReopen states the contradiction that reopened this phase. Without it
|
||||
// the planning session a replan convenes gets the ordinary plan brief and an
|
||||
// empty template, with nothing saying a plan already exists, what contradicted
|
||||
// it, or that its output supersedes an accepted plan (F66). Everything here is
|
||||
// durable on PlanMismatchRecorded; it simply never reached the agent.
|
||||
//
|
||||
// Only the phases a mismatch can reopen render it. An implementer that finds
|
||||
// the next contradiction is looking at the code, not at this account of the
|
||||
// last one.
|
||||
func renderReopen(in Input) string {
|
||||
m := in.Task.PlanMismatch
|
||||
if m == nil || (in.Phase != domain.WorkPhasePlan && in.Phase != domain.WorkPhaseResearch) {
|
||||
return ""
|
||||
}
|
||||
var b strings.Builder
|
||||
b.WriteString("\n## Why this phase reopened\n\n")
|
||||
fmt.Fprintf(&b, "A plan was already accepted and the code contradicted it. Orchestra reopened this phase to settle that, and the session that found it is gone.\n\n")
|
||||
fmt.Fprintf(&b, "- phase: %s\n", collapse(m.PhaseID))
|
||||
fmt.Fprintf(&b, "- observed: %s\n", collapse(m.Observed))
|
||||
fmt.Fprintf(&b, "- the plan says: %s\n", collapse(m.Contradicts))
|
||||
for _, e := range m.Evidence {
|
||||
fmt.Fprintf(&b, "- evidence: %s\n", collapse(e))
|
||||
}
|
||||
if in.Phase == domain.WorkPhasePlan {
|
||||
b.WriteString("\nThe accepted plan stays accepted until you seal a replacement, and sealing one supersedes it along with every phase it had verified. Address the contradiction above: a replacement that repeats it will be contradicted again.\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func renderSealed(in Input) string {
|
||||
var b strings.Builder
|
||||
research := in.Research
|
||||
|
||||
@@ -734,3 +734,61 @@ func TestPendingManualGateRendersStale(t *testing.T) {
|
||||
t.Errorf("a pending manual gate at HEAD rendered stale:\n%s", fresh)
|
||||
}
|
||||
}
|
||||
|
||||
// F66, found live on run 20. A replan convenes a planning session to settle a
|
||||
// contradiction, and that session was told nothing about it: the ordinary plan
|
||||
// brief, an empty template, and no mention that a plan already exists.
|
||||
func TestAReopenedPlanPhaseIsToldWhatContradictedThePlan(t *testing.T) {
|
||||
in := input()
|
||||
in.Phase = domain.WorkPhasePlan
|
||||
in.Task.PlanRef = "plan-a"
|
||||
in.Task.PlanMismatch = &domain.PlanMismatch{
|
||||
PlanRef: "plan-a", PhaseID: "phase-3", AtSHA: "18ccaf00000000000000000000000000000000aa",
|
||||
Observed: "the aggregation runs per figure, not per person",
|
||||
Contradicts: "the plan states the pipeline already aggregates per person",
|
||||
Evidence: []string{"internal/figures/aggregate.go:88"},
|
||||
RequestedAction: domain.PlanMismatchReplan,
|
||||
}
|
||||
out, err := Build(in)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"## Why this phase reopened",
|
||||
"phase: phase-3",
|
||||
"the aggregation runs per figure",
|
||||
"the plan states the pipeline already aggregates per person",
|
||||
"internal/figures/aggregate.go:88",
|
||||
"stays accepted until you seal a replacement",
|
||||
} {
|
||||
if !strings.Contains(out.Task, want) {
|
||||
t.Fatalf("the planner was not told %q:\n%s", want, out.Task)
|
||||
}
|
||||
}
|
||||
// It must come before the artifacts it changes the reading of.
|
||||
in.Research = &workphase.Research{Findings: []workphase.Finding{{
|
||||
ID: "r1", Confidence: "fact", Claim: "aggregation is per figure",
|
||||
Evidence: "internal/figures/aggregate.go:88",
|
||||
}}}
|
||||
out, err = Build(in)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(out.Task, "## Accepted research") {
|
||||
t.Fatal("the research this planner works from is missing")
|
||||
}
|
||||
if strings.Index(out.Task, "## Why this phase reopened") > strings.Index(out.Task, "## Accepted research") {
|
||||
t.Fatal("the reason this session exists is rendered below the material it qualifies")
|
||||
}
|
||||
|
||||
// An implementer is looking at the code, not at an account of the last
|
||||
// contradiction, so it renders nowhere else.
|
||||
in.Phase = domain.WorkPhaseImplement
|
||||
out, err = Build(in)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(out.Task, "## Why this phase reopened") {
|
||||
t.Fatal("a settled contradiction reached the implementation phase")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -113,6 +113,12 @@ type DebtObservation struct {
|
||||
Detail string `json:"detail,omitempty"`
|
||||
Paths []string `json:"paths,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
// Repeats is how many times this one incident recurred. It is intensity,
|
||||
// never recurrence: one worker stuck in a five-second retry loop produced
|
||||
// 301 repeats of a single failure, and counting those as 301 pieces of
|
||||
// evidence would make one broken worker look like chronic, system-wide
|
||||
// debt. Recurrence is the number of independent observations.
|
||||
Repeats int `json:"repeats,omitempty"`
|
||||
}
|
||||
|
||||
func (o DebtObservation) Validate() error {
|
||||
@@ -251,7 +257,7 @@ func DebtClassForBlockReason(r BlockReason) (DebtClass, bool) {
|
||||
// classes a worker actually emits are listed; an unknown one is not guessed at.
|
||||
func DebtClassForFailureClass(f string) (DebtClass, bool) {
|
||||
switch f {
|
||||
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired":
|
||||
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired", "handoff_unanswered":
|
||||
return DebtOperational, true
|
||||
case "invalid_handoff":
|
||||
return DebtCorrectness, true
|
||||
|
||||
@@ -82,6 +82,23 @@ const (
|
||||
BlockReasonUnknown BlockReason = "unknown"
|
||||
)
|
||||
|
||||
// BlockReasonAwaitsReply reports whether this stop is a question waiting on a
|
||||
// human, rather than a fault or a budget. Every reason listed here resumes on
|
||||
// a reply, so both the loop that goes and reads replies and the loop that
|
||||
// returns answered tasks to the queue must agree on the set. They did not:
|
||||
// plan_mismatch blocked for an answer that nothing ever went to fetch, and
|
||||
// nothing would have resumed it if it had (F64).
|
||||
//
|
||||
// operator_required is deliberately absent. A task that has spent its question
|
||||
// budget continues when an operator says so, not when someone replies.
|
||||
func BlockReasonAwaitsReply(r BlockReason) bool {
|
||||
switch r {
|
||||
case BlockReasonHumanDecision, BlockReasonTrajectoryGate, BlockReasonPlanMismatch:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (r BlockReason) Valid() bool {
|
||||
switch r {
|
||||
case BlockReasonLeaseFailure, BlockReasonWorkerOffline, BlockReasonLeaseExpired,
|
||||
@@ -168,19 +185,19 @@ type Task struct {
|
||||
QualityGate string `json:"quality_gate,omitempty"`
|
||||
// Block evidence is projected from TaskBlocked so terminal records remain
|
||||
// diagnosable after the live coordinator mapping is gone.
|
||||
Blocker string `json:"blocker,omitempty"`
|
||||
BlockReason BlockReason `json:"block_reason,omitempty"`
|
||||
BlockedAt time.Time `json:"blocked_at,omitempty"`
|
||||
LastPaneID string `json:"last_pane_id,omitempty"`
|
||||
LastHarness string `json:"last_harness_id,omitempty"`
|
||||
Blocker string `json:"blocker,omitempty"`
|
||||
BlockReason BlockReason `json:"block_reason,omitempty"`
|
||||
BlockedAt time.Time `json:"blocked_at,omitempty"`
|
||||
LastPaneID string `json:"last_pane_id,omitempty"`
|
||||
LastHarness string `json:"last_harness_id,omitempty"`
|
||||
// LastLeaseEpoch is the fencing token of the lease that most recently
|
||||
// ended. A worker can push its release anchor and only then discover the
|
||||
// lease expired; the finished work is durable in git but the commit can
|
||||
// never land. Retaining the epoch lets exactly that owner still commit
|
||||
// while the task sits unleased.
|
||||
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
|
||||
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
|
||||
LastSession SessionEvidence `json:"last_session,omitempty"`
|
||||
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
|
||||
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
|
||||
LastSession SessionEvidence `json:"last_session,omitempty"`
|
||||
// Recovery state is part of the durable projection, never process-local
|
||||
// router memory. This makes retry and operator diagnostics survive a
|
||||
// coordinator restart.
|
||||
@@ -217,6 +234,13 @@ type Task struct {
|
||||
// current one, oldest first. A superseded plan stays queryable: the
|
||||
// verification recorded against it is provenance, not garbage.
|
||||
PlanHistory []string `json:"plan_history,omitempty"`
|
||||
// PlanMismatch is the contradiction that reopened this task's phase. The
|
||||
// planning session convened by a replan has to be told what it is there to
|
||||
// fix, and the implementer that found it is gone by then (F66). It is
|
||||
// cleared when a replacement plan is accepted, or when the stop it caused
|
||||
// is answered, because after either it is history rather than a live
|
||||
// instruction.
|
||||
PlanMismatch *PlanMismatch `json:"plan_mismatch,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
@@ -269,6 +293,25 @@ func NewID() string {
|
||||
_, _ = rand.Read(b[6:])
|
||||
return ulidEncoding.EncodeToString(b)
|
||||
}
|
||||
|
||||
// EventWithoutTask reports whether an event records something about the
|
||||
// system rather than about one task's lifecycle. These carry the "system"
|
||||
// aggregate id, so no task projection has to exist for them.
|
||||
//
|
||||
// One list, because there were three: the replay guard, the append guard and
|
||||
// the transition check each kept their own copy, and adding an event type to
|
||||
// two of them left it rejected by the third.
|
||||
func EventWithoutTask(typ string) bool {
|
||||
switch typ {
|
||||
case "QuotaReported", "StandupAdvisory", "ApprovalGranted", "ApprovalDenied",
|
||||
EventObservationIncidentOpened, EventObservationIncidentClosed:
|
||||
// An intervention is deliberately absent: it names a task when it
|
||||
// repaired one, and that task must exist.
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func ValidateEvent(e Event) error {
|
||||
if e.SchemaVersion > CurrentEventSchema || e.Type == "" || e.TaskID == "" || len(e.Payload) == 0 || len(e.Payload) > 64*1024 {
|
||||
return ErrInvalid
|
||||
@@ -276,7 +319,7 @@ func ValidateEvent(e Event) error {
|
||||
if e.SchemaVersion >= 2 && strings.TrimSpace(e.Surface) == "" {
|
||||
return fmt.Errorf("%w: surface required", ErrInvalid)
|
||||
}
|
||||
allowed := map[string]bool{"TaskCreated": true, "TaskLeased": true, "TaskLeaseRenewed": true, "TaskReleased": true, "TaskLaunchAcknowledged": true, "TaskPickupValidated": true, "TaskCompleted": true, "TaskFailed": true, "TaskBlocked": true, "TaskNeedsAttention": true, "ApprovalRequested": true, "ApprovalGranted": true, "ApprovalDenied": true, "TaskAmended": true, "TaskCorrected": true, "QuotaReported": true, "StandupAdvisory": true, EventHumanDecisionRecorded: true, EventHumanDecisionSuperseded: true, EventWorkPhaseChanged: true, EventDeferredFindingRecorded: true, EventReviewRecorded: true, EventTaskSubmitted: true, EventTaskChangesRequested: true, EventPlanPhaseVerified: true, EventPlanMismatchRecorded: true}
|
||||
allowed := map[string]bool{"TaskCreated": true, "TaskLeased": true, "TaskLeaseRenewed": true, "TaskReleased": true, "TaskLaunchAcknowledged": true, "TaskPickupValidated": true, "TaskCompleted": true, "TaskFailed": true, "TaskBlocked": true, "TaskNeedsAttention": true, "ApprovalRequested": true, "ApprovalGranted": true, "ApprovalDenied": true, "TaskAmended": true, "TaskCorrected": true, "QuotaReported": true, "StandupAdvisory": true, EventHumanDecisionRecorded: true, EventHumanDecisionSuperseded: true, EventWorkPhaseChanged: true, EventDeferredFindingRecorded: true, EventReviewRecorded: true, EventTaskSubmitted: true, EventTaskChangesRequested: true, EventPlanPhaseVerified: true, EventPlanMismatchRecorded: true, EventObservationIncidentOpened: true, EventObservationIncidentClosed: true, EventOperatorInterventionRecorded: true}
|
||||
if !allowed[e.Type] {
|
||||
return fmt.Errorf("%w: unknown type %q", ErrInvalid, e.Type)
|
||||
}
|
||||
@@ -558,6 +601,12 @@ func ValidatePayload(typ string, p map[string]any) error {
|
||||
return ValidatePlanPhaseVerified(p)
|
||||
case EventPlanMismatchRecorded:
|
||||
return ValidatePlanMismatchRecorded(p)
|
||||
case EventObservationIncidentOpened:
|
||||
return ValidateObservationIncidentOpened(p)
|
||||
case EventObservationIncidentClosed:
|
||||
return ValidateObservationIncidentClosed(p)
|
||||
case EventOperatorInterventionRecorded:
|
||||
return ValidateOperatorInterventionRecorded(p)
|
||||
case EventReviewRecorded:
|
||||
if err := requiredHash(p, "artifact_ref"); err != nil {
|
||||
return err
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
package domain
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// EventOperatorInterventionRecorded is a human saying what they repaired by
|
||||
// hand. Nothing infers it: a manual fix happens outside Orchestra by
|
||||
// definition, so the only honest way to have the evidence is for the operator
|
||||
// to state it. Without this the debt ledger reported manual recovery as a
|
||||
// permanent gap, and every repair that kept the system running was invisible
|
||||
// to the record of how much the system costs to run.
|
||||
const EventOperatorInterventionRecorded = "OperatorInterventionRecorded"
|
||||
|
||||
// InterventionKind is what the operator did. The list is closed so the ledger
|
||||
// can group repairs; an unrecognised kind is refused rather than guessed at.
|
||||
type InterventionKind string
|
||||
|
||||
const (
|
||||
InterventionWorkerRestart InterventionKind = "worker_restart"
|
||||
InterventionTransactionClean InterventionKind = "transaction_cleanup"
|
||||
InterventionForcedRelease InterventionKind = "forced_release"
|
||||
InterventionStateRepair InterventionKind = "state_repair"
|
||||
InterventionManualRequeue InterventionKind = "manual_requeue"
|
||||
InterventionPhaseRecovery InterventionKind = "manual_phase_recovery"
|
||||
)
|
||||
|
||||
func (k InterventionKind) Valid() bool {
|
||||
switch k {
|
||||
case InterventionWorkerRestart, InterventionTransactionClean, InterventionForcedRelease,
|
||||
InterventionStateRepair, InterventionManualRequeue, InterventionPhaseRecovery:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// OperatorIntervention is one recorded manual repair.
|
||||
type OperatorIntervention struct {
|
||||
TaskID string `json:"task_id,omitempty"`
|
||||
WorkerID string `json:"worker_id,omitempty"`
|
||||
Kind InterventionKind `json:"kind"`
|
||||
// Reason is the operator's own account of why it was needed. It is the
|
||||
// part a later reader cannot reconstruct from anything else.
|
||||
Reason string `json:"reason"`
|
||||
// RelatedEventID and RelatedTransactionID point at what was repaired, so a
|
||||
// reader can find the failure this answered rather than infer it.
|
||||
RelatedEventID string `json:"related_event_id,omitempty"`
|
||||
RelatedTransactionID string `json:"related_transaction_id,omitempty"`
|
||||
Components []string `json:"components,omitempty"`
|
||||
At time.Time `json:"at,omitempty"`
|
||||
}
|
||||
|
||||
const maxInterventionReason = 1000
|
||||
|
||||
func (i OperatorIntervention) Validate() error {
|
||||
if !i.Kind.Valid() {
|
||||
return fmt.Errorf("%w: %q is not an intervention kind", ErrInvalid, i.Kind)
|
||||
}
|
||||
if strings.TrimSpace(i.Reason) == "" {
|
||||
return fmt.Errorf("%w: an intervention states why it was needed", ErrInvalid)
|
||||
}
|
||||
if len(i.Reason) > maxInterventionReason {
|
||||
return fmt.Errorf("%w: reason exceeds %d characters", ErrInvalid, maxInterventionReason)
|
||||
}
|
||||
if i.TaskID == "" && i.WorkerID == "" {
|
||||
return fmt.Errorf("%w: an intervention names the task or the worker it repaired", ErrInvalid)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateOperatorInterventionRecorded(p map[string]any) error {
|
||||
kind, _ := p["kind"].(string)
|
||||
reason, _ := p["reason"].(string)
|
||||
task, _ := p["task_id"].(string)
|
||||
worker, _ := p["worker_id"].(string)
|
||||
return OperatorIntervention{
|
||||
Kind: InterventionKind(kind), Reason: reason, TaskID: task, WorkerID: worker,
|
||||
}.Validate()
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
package domain
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A worker's observation ring is bounded, lossy, and local: it holds distinct
|
||||
// failure messages with repeat counts and nothing else, and it disappears when
|
||||
// the process does. The debt ledger reported that gap about itself, because no
|
||||
// event carried any of it.
|
||||
//
|
||||
// These two events make it durable as incidents rather than as symptoms. Run
|
||||
// 11 saw the same 409 refusal 301 times; that is one incident with an
|
||||
// intensity of 301, not 301 pieces of evidence. Recurrence has to mean "this
|
||||
// happened on four independent leases", or one stuck loop makes everything
|
||||
// look chronic.
|
||||
const (
|
||||
EventObservationIncidentOpened = "ObservationIncidentOpened"
|
||||
EventObservationIncidentClosed = "ObservationIncidentClosed"
|
||||
)
|
||||
|
||||
// ObservationCloseReason is why Orchestra finalized an incident. None of them
|
||||
// is "the message stopped appearing in the ring": the ring is a bounded
|
||||
// history, so absence proves eviction as easily as recovery.
|
||||
type ObservationCloseReason string
|
||||
|
||||
const (
|
||||
// ObservationCloseLeaseEnd and ObservationCloseEpochChange are the natural
|
||||
// boundaries of a lease-scoped incident. The work it was about is over.
|
||||
ObservationCloseLeaseEnd ObservationCloseReason = "lease_end"
|
||||
ObservationCloseEpochChange ObservationCloseReason = "epoch_change"
|
||||
// ObservationCloseWorkerRestart ends every incident of an incarnation. A
|
||||
// new process cannot continue the old one's symptom.
|
||||
ObservationCloseWorkerRestart ObservationCloseReason = "worker_restart"
|
||||
// ObservationCloseQuietTimeout is the only closer for an observation with
|
||||
// no lease to bound it, and it fires on last_seen going stale rather than
|
||||
// on the entry vanishing.
|
||||
ObservationCloseQuietTimeout ObservationCloseReason = "quiet_timeout"
|
||||
)
|
||||
|
||||
func (r ObservationCloseReason) Valid() bool {
|
||||
switch r {
|
||||
case ObservationCloseLeaseEnd, ObservationCloseEpochChange, ObservationCloseWorkerRestart, ObservationCloseQuietTimeout:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// WorkerObservation is one entry of a worker's ring as reported on a
|
||||
// heartbeat. It is the input to the incident projection, never a stored event.
|
||||
type WorkerObservation struct {
|
||||
Message string `json:"message"`
|
||||
Count int `json:"count"`
|
||||
First time.Time `json:"first"`
|
||||
Last time.Time `json:"last"`
|
||||
}
|
||||
|
||||
// ObservationIncident is one durable incident: a signature seen by one worker,
|
||||
// on one lease when there is one, from its first occurrence to the boundary
|
||||
// that ended it.
|
||||
type ObservationIncident struct {
|
||||
ID string `json:"observation_id"`
|
||||
WorkerID string `json:"worker_id"`
|
||||
Incarnation string `json:"incarnation,omitempty"`
|
||||
TaskID string `json:"task_id,omitempty"`
|
||||
LeaseEpoch string `json:"lease_epoch,omitempty"`
|
||||
// Signature is the message with its task ids, commit shas, paths and
|
||||
// durations replaced, so the same failure on two tasks shares it. Grouping
|
||||
// on the raw message would make every task its own kind of problem.
|
||||
Signature string `json:"signature"`
|
||||
Detail string `json:"detail,omitempty"`
|
||||
FirstSeen time.Time `json:"first_seen"`
|
||||
// LastSeen is the last actual occurrence. ClosedAt is when Orchestra
|
||||
// finalized the incident, which is later and often much later: an incident
|
||||
// stays open until its lease ends, and open means "not yet final evidence"
|
||||
// rather than "happening right now".
|
||||
LastSeen time.Time `json:"last_seen,omitempty"`
|
||||
ClosedAt time.Time `json:"closed_at,omitempty"`
|
||||
RepeatCount int `json:"repeat_count,omitempty"`
|
||||
CloseReason ObservationCloseReason `json:"close_reason,omitempty"`
|
||||
}
|
||||
|
||||
// Key identifies an incident. Two workers reporting the same failure are two
|
||||
// incidents, and so are two leases of one task.
|
||||
func (i ObservationIncident) Key() string {
|
||||
return strings.Join([]string{i.WorkerID, i.TaskID, i.LeaseEpoch, i.Signature}, "\x00")
|
||||
}
|
||||
|
||||
var (
|
||||
// Case-insensitive: a task id appears upper-case in a message and
|
||||
// lower-case inside a pane name, and the live ledger's first run showed
|
||||
// pane names keeping their task, which would give the same failure a
|
||||
// different signature on every task.
|
||||
observationID = regexp.MustCompile(`(?i)\b[0-9A-HJKMNP-TV-Z]{26}\b`)
|
||||
observationSHA = regexp.MustCompile(`\b[0-9a-f]{7,64}\b`)
|
||||
observationDuration = regexp.MustCompile(`\b\d+(\.\d+)?(ns|us|µs|ms|s|m|h)(\d+(\.\d+)?(ns|us|µs|ms|s|m|h))*\b`)
|
||||
observationNumber = regexp.MustCompile(`\b\d+\b`)
|
||||
observationPath = regexp.MustCompile(`(/[\w.-]+){2,}`)
|
||||
)
|
||||
|
||||
// ObservationSignature collapses one message to the kind of failure it is.
|
||||
// "lease A not renewed" and "lease B not renewed" are the same problem seen
|
||||
// twice, which is the whole basis of counting recurrence across tasks.
|
||||
func ObservationSignature(message string) string {
|
||||
s := strings.TrimSpace(message)
|
||||
s = observationID.ReplaceAllString(s, "<id>")
|
||||
s = observationPath.ReplaceAllString(s, "<path>")
|
||||
s = observationDuration.ReplaceAllString(s, "<dur>")
|
||||
s = observationSHA.ReplaceAllString(s, "<sha>")
|
||||
s = observationNumber.ReplaceAllString(s, "<n>")
|
||||
s = strings.Join(strings.Fields(s), " ")
|
||||
if len(s) > 200 {
|
||||
s = s[:200]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// ObservationTaskID reads the task a failure was about out of the message
|
||||
// itself. The ring is a history: it holds entries from tasks that ended long
|
||||
// ago, so the worker's currently active task is the wrong answer for most of
|
||||
// them, and attributing an old failure to whatever is running now would be a
|
||||
// fabricated association.
|
||||
func ObservationTaskID(message string) string {
|
||||
if m := observationTaskID.FindString(message); m != "" {
|
||||
return strings.ToUpper(m)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
var observationTaskID = regexp.MustCompile(`(?i)\b[0-9A-HJKMNP-TV-Z]{26}\b`)
|
||||
|
||||
func ValidateObservationIncidentOpened(p map[string]any) error {
|
||||
if id, _ := p["observation_id"].(string); strings.TrimSpace(id) == "" {
|
||||
return fmt.Errorf("%w: observation_id required", ErrInvalid)
|
||||
}
|
||||
if w, _ := p["worker_id"].(string); strings.TrimSpace(w) == "" {
|
||||
return fmt.Errorf("%w: worker_id required", ErrInvalid)
|
||||
}
|
||||
if sig, _ := p["signature"].(string); strings.TrimSpace(sig) == "" {
|
||||
return fmt.Errorf("%w: signature required", ErrInvalid)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateObservationIncidentClosed(p map[string]any) error {
|
||||
if id, _ := p["observation_id"].(string); strings.TrimSpace(id) == "" {
|
||||
return fmt.Errorf("%w: observation_id required", ErrInvalid)
|
||||
}
|
||||
reason, _ := p["close_reason"].(string)
|
||||
if !ObservationCloseReason(reason).Valid() {
|
||||
return fmt.Errorf("%w: close_reason %q is not a close reason", ErrInvalid, reason)
|
||||
}
|
||||
if c, ok := p["repeat_count"].(float64); ok && c < 0 {
|
||||
return fmt.Errorf("%w: repeat_count cannot be negative", ErrInvalid)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -55,6 +55,13 @@ type PlanPhaseRecord struct {
|
||||
// EvidenceRef is the CAS ref of the captured command output.
|
||||
EvidenceRef string `json:"evidence_ref,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
// ManualAtSHA is the tree a human was actually looking at when they signed
|
||||
// this phase off. A manual check on most projects is a human reading
|
||||
// output, so a sign-off establishes something about one tree and nothing
|
||||
// about the next one (F63). Rerunning the automated half re-establishes it
|
||||
// at the new commit; the manual half has to be given again, and this is
|
||||
// what makes the difference visible instead of assumed.
|
||||
ManualAtSHA string `json:"manual_at_sha,omitempty"`
|
||||
}
|
||||
|
||||
// Stale reports whether the tree has moved since this phase was verified. A
|
||||
|
||||
@@ -21,10 +21,14 @@ type Worker struct {
|
||||
Capacity int `json:"capacity"`
|
||||
SupportedProjects []string `json:"supported_projects"`
|
||||
Build buildinfo.Info `json:"build"`
|
||||
LastSeen time.Time `json:"last_seen"`
|
||||
Online bool `json:"online"`
|
||||
Health WorkerHealth `json:"health"`
|
||||
Token string `json:"-"`
|
||||
// Incarnation identifies one worker process. Nothing else on the wire
|
||||
// distinguishes a restarted worker from a running one, and an observation
|
||||
// incident cannot outlive the process that reported it.
|
||||
Incarnation string `json:"incarnation,omitempty"`
|
||||
LastSeen time.Time `json:"last_seen"`
|
||||
Online bool `json:"online"`
|
||||
Health WorkerHealth `json:"health"`
|
||||
Token string `json:"-"`
|
||||
}
|
||||
|
||||
// WorkerHealth is reported by the worker that owns the local execution backend.
|
||||
@@ -42,6 +46,9 @@ type WorkerHealth struct {
|
||||
// Observations is the bounded set of distinct failures behind LastError,
|
||||
// which keeps its wire name and still reports only the newest.
|
||||
Observations []Observation `json:"observations,omitempty"`
|
||||
// Incarnation repeats the worker's process identity on every heartbeat, so
|
||||
// the coordinator sees a restart even if it missed the registration.
|
||||
Incarnation string `json:"incarnation,omitempty"`
|
||||
}
|
||||
|
||||
// Observation is one distinct worker failure with its repeat count. A single
|
||||
|
||||
@@ -174,6 +174,14 @@ type Session struct {
|
||||
// its §6.1 handoff (HandoffFile) — avoids re-sending the same prompt
|
||||
// every tick while Release keeps waiting for the file to appear.
|
||||
HandoffRequested bool `json:"handoff_requested,omitempty"`
|
||||
// HandoffRequestedAt stamps that prompt. A request nobody answers used to
|
||||
// end as an ordinary idle expiry, indistinguishable from an agent that
|
||||
// never started (F62); the stamp is what makes the wait bounded and the
|
||||
// giving-up causal.
|
||||
HandoffRequestedAt time.Time `json:"handoff_requested_at,omitempty"`
|
||||
// HandoffRetried records that the request was re-sent once, so a session
|
||||
// waiting on an answer is not re-prompted every tick.
|
||||
HandoffRetried bool `json:"handoff_retried,omitempty"`
|
||||
// HandoffReason is selected by the coordinator when it asks for the
|
||||
// semantic report. The checkout worker, rather than the harness, copies
|
||||
// it into the canonical handoff it seals at release time.
|
||||
|
||||
@@ -43,9 +43,17 @@ func CheckDebtEligibility(item domain.DebtItem) DebtCheck {
|
||||
if len(why) > 0 {
|
||||
return DebtCheck{true, why}
|
||||
}
|
||||
// Reasons state facts about this item and nothing about the
|
||||
// architecture around it. The second line here used to say manual
|
||||
// interventions were recorded by no event type, which was true when it
|
||||
// was written and false the moment OperatorInterventionRecorded
|
||||
// landed. A decision that is right for a reason that has become a lie
|
||||
// cannot be audited, and why a count is zero is not this function's
|
||||
// business: no intervention happened, none was recorded, or none was
|
||||
// migrated all read the same from here.
|
||||
return DebtCheck{false, []string{
|
||||
fmt.Sprintf("needs 3 occurrences across 2 tasks, or 1 manual intervention; has %d across %d tasks with %d interventions", recurrence, tasks, manual),
|
||||
"manual interventions are not recorded by any event type, so that count reads 0 on every current log",
|
||||
fmt.Sprintf("breadth threshold not met: %d occurrences across %d tasks, needs 3 across 2", recurrence, tasks),
|
||||
fmt.Sprintf("manual intervention threshold not met: %d recorded, needs 1", manual),
|
||||
}}
|
||||
case domain.DebtStructural:
|
||||
if recurrence >= 3 {
|
||||
|
||||
@@ -62,7 +62,17 @@ func TestDebtRefusalNamesTheMissingEvidence(t *testing.T) {
|
||||
t.Fatalf("want a refusal with reasons, got %+v", check)
|
||||
}
|
||||
joined := strings.Join(check.Reasons, " ")
|
||||
if !strings.Contains(joined, "manual interventions are not recorded") {
|
||||
t.Fatalf("the refusal must say the intervention count is structurally zero: %v", check.Reasons)
|
||||
// Every reason is a fact about this item. The old text asserted that no
|
||||
// event type recorded manual interventions, which stopped being true the
|
||||
// day one did, leaving a correct decision explained by a lie.
|
||||
for _, want := range []string{"1 occurrences across 1 tasks", "0 recorded, needs 1"} {
|
||||
if !strings.Contains(joined, want) {
|
||||
t.Fatalf("the refusal does not state %q: %v", want, check.Reasons)
|
||||
}
|
||||
}
|
||||
for _, forbidden := range []string{"event type", "every current log"} {
|
||||
if strings.Contains(joined, forbidden) {
|
||||
t.Fatalf("a reason describes the architecture instead of the item: %v", check.Reasons)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -58,18 +58,27 @@ func RequestHumanDecision(s *store.Store, project registry.Project, taskID strin
|
||||
return blockTask(s, t, domain.BlockReasonHumanDecision, req.Render(), &req)
|
||||
}
|
||||
|
||||
// fenceToLease binds a lifecycle event to the lease that is producing it.
|
||||
// Store.Append fences every lifecycle event on a leased task against the
|
||||
// current owner and epoch, so an event that omits them is a conflict rather
|
||||
// than a block. Every coordinator-side stop goes through here: two of them did
|
||||
// not, and both failed silently against a live lease (F65). A task stops for a
|
||||
// human only while some session is running, so the leased case is the only one
|
||||
// that ever mattered.
|
||||
func fenceToLease(payload map[string]any, t domain.Task) {
|
||||
if t.Lease == nil {
|
||||
return
|
||||
}
|
||||
payload["harness_id"] = t.Lease.HarnessID
|
||||
payload["lease_epoch"] = t.Lease.Epoch
|
||||
}
|
||||
|
||||
func blockTask(s *store.Store, t domain.Task, reason domain.BlockReason, blocker string, req *domain.DecisionRequest) (domain.Event, error) {
|
||||
payload := map[string]any{
|
||||
"blocker": blocker, "block_reason": string(reason),
|
||||
"lifecycle_phase": "awaiting_human",
|
||||
}
|
||||
if t.Lease != nil {
|
||||
// Store.Append fences every lifecycle event on a leased task against
|
||||
// the current owner and epoch. A question from a session that no
|
||||
// longer owns the task is a conflict, not a block.
|
||||
payload["harness_id"] = t.Lease.HarnessID
|
||||
payload["lease_epoch"] = t.Lease.Epoch
|
||||
}
|
||||
fenceToLease(payload, t)
|
||||
if req != nil {
|
||||
payload["decision_request"] = req
|
||||
}
|
||||
@@ -107,11 +116,9 @@ func ResumeAnsweredBlockers(s *store.Store) ([]domain.Event, error) {
|
||||
if t.State != domain.StateBlocked {
|
||||
continue
|
||||
}
|
||||
switch t.BlockReason {
|
||||
case domain.BlockReasonHumanDecision, domain.BlockReasonTrajectoryGate:
|
||||
default:
|
||||
// operator_required is deliberately not resumed by a reply. An
|
||||
// operator decides when a task that spent its budget continues.
|
||||
// operator_required is deliberately not resumed by a reply. An
|
||||
// operator decides when a task that spent its budget continues.
|
||||
if !domain.BlockReasonAwaitsReply(t.BlockReason) {
|
||||
continue
|
||||
}
|
||||
if !blockerAnswered(s, t.ID, t.BlockReason) {
|
||||
|
||||
@@ -0,0 +1,275 @@
|
||||
package operations
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"orchestra/internal/authz"
|
||||
"orchestra/internal/domain"
|
||||
"orchestra/internal/store"
|
||||
)
|
||||
|
||||
// QuietTimeout bounds an incident that has no lease to bound it. A worker-level
|
||||
// observation has no terminal boundary of its own, so staleness of its last
|
||||
// actual occurrence is the only honest closer.
|
||||
const QuietTimeout = 10 * time.Minute
|
||||
|
||||
// ObservationTracker turns a worker's bounded, lossy ring into durable
|
||||
// incidents. It is deliberately not a copy of the ring.
|
||||
//
|
||||
// The rules that matter, and why:
|
||||
//
|
||||
// - An incident is opened at first sight and appended immediately, so a
|
||||
// coordinator that dies mid-incident still leaves the fact that it existed.
|
||||
// - A repeat updates the aggregate and appends nothing. Run 11's 409 loop
|
||||
// repeated 301 times; appending each would have been 301 pieces of evidence
|
||||
// for one problem, and would have made every debt item eligible at once.
|
||||
// - Absence from the ring closes nothing. The ring is a bounded history, so a
|
||||
// message can vanish because it was evicted rather than because it stopped.
|
||||
// - The lease is the scope. The same signature going quiet and returning
|
||||
// inside one epoch is one incident, not two recurrences.
|
||||
type ObservationTracker struct {
|
||||
Store *store.Store
|
||||
// counts is the last count this tracker saw for an open incident, so a
|
||||
// ring entry that is evicted and recreated accumulates rather than
|
||||
// restarting. Reported 34, evicted, reported 3 again means 37 occurrences,
|
||||
// not 3. In-memory: a restart loses the accumulation, never the incident.
|
||||
counts map[string]int
|
||||
// seen is the high-water mark per worker and signature, kept after an
|
||||
// incident closes. The ring is a history, so a closed incident's entry
|
||||
// keeps being reported for as long as it survives eviction; without this,
|
||||
// every heartbeat after a quiet timeout opened the same incident again and
|
||||
// manufactured recurrence out of one old failure. Live on the first real
|
||||
// run: three signatures, four incidents each, none of them a new event.
|
||||
seen map[string]watermark
|
||||
// incarnations is the last incarnation seen per worker, which is what makes
|
||||
// a restart detectable at all.
|
||||
incarnations map[string]string
|
||||
}
|
||||
|
||||
// watermark is the last occurrence this tracker accounted for one worker's
|
||||
// signature, whether or not its incident is still open.
|
||||
type watermark struct {
|
||||
last time.Time
|
||||
count int
|
||||
}
|
||||
|
||||
// WorkerReport is one heartbeat's worth of attributed observations. The
|
||||
// coordinator attributes them, because the ring carries only messages: the
|
||||
// worker's active task and that task's current lease epoch are what bind an
|
||||
// incident to the work it happened during.
|
||||
type WorkerReport struct {
|
||||
WorkerID string
|
||||
Incarnation string
|
||||
TaskID string
|
||||
LeaseEpoch string
|
||||
Observations []domain.WorkerObservation
|
||||
At time.Time
|
||||
}
|
||||
|
||||
// Ingest folds one heartbeat into the durable incidents and returns the events
|
||||
// it appended. Every close it decides is one of the four boundaries; none of
|
||||
// them is "the message is no longer in the ring".
|
||||
func (t *ObservationTracker) Ingest(r WorkerReport) ([]domain.Event, error) {
|
||||
if t == nil || t.Store == nil || r.WorkerID == "" {
|
||||
return nil, nil
|
||||
}
|
||||
if t.counts == nil {
|
||||
t.counts, t.incarnations = map[string]int{}, map[string]string{}
|
||||
t.seen = map[string]watermark{}
|
||||
}
|
||||
at := r.At
|
||||
if at.IsZero() {
|
||||
at = time.Now().UTC()
|
||||
}
|
||||
var appended []domain.Event
|
||||
|
||||
// A new process cannot continue the previous one's symptom, so its
|
||||
// incidents are finalized before anything this heartbeat says is folded in.
|
||||
if previous, seen := t.incarnations[r.WorkerID]; r.Incarnation != "" && seen && previous != r.Incarnation {
|
||||
closed, err := t.closeWhere(at, domain.ObservationCloseWorkerRestart, func(inc domain.ObservationIncident) bool {
|
||||
return inc.WorkerID == r.WorkerID
|
||||
})
|
||||
appended = append(appended, closed...)
|
||||
if err != nil {
|
||||
return appended, err
|
||||
}
|
||||
}
|
||||
if r.Incarnation != "" {
|
||||
t.incarnations[r.WorkerID] = r.Incarnation
|
||||
}
|
||||
|
||||
open := map[string]domain.ObservationIncident{}
|
||||
for _, inc := range t.Store.OpenObservations() {
|
||||
if inc.WorkerID == r.WorkerID {
|
||||
open[inc.Key()] = inc
|
||||
}
|
||||
}
|
||||
for _, o := range r.Observations {
|
||||
signature := domain.ObservationSignature(o.Message)
|
||||
if signature == "" {
|
||||
continue
|
||||
}
|
||||
// The task comes from the message, because the ring outlives the work
|
||||
// it describes. Only a failure that names no task is attributed to the
|
||||
// lease running now.
|
||||
taskID, epoch := domain.ObservationTaskID(o.Message), ""
|
||||
if taskID == "" {
|
||||
taskID, epoch = r.TaskID, r.LeaseEpoch
|
||||
} else if taskID == r.TaskID {
|
||||
epoch = r.LeaseEpoch
|
||||
}
|
||||
candidate := domain.ObservationIncident{
|
||||
WorkerID: r.WorkerID, Incarnation: r.Incarnation, Signature: signature,
|
||||
TaskID: taskID, LeaseEpoch: epoch,
|
||||
}
|
||||
mark := t.seen[r.WorkerID+"\x00"+signature]
|
||||
existing, isOpen := open[candidate.Key()]
|
||||
if !isOpen {
|
||||
// Nothing new: this is a closed incident's entry still sitting in
|
||||
// the ring. Presence is not occurrence.
|
||||
if !firstOr(o.Last, at).After(mark.last) && o.Count <= mark.count {
|
||||
continue
|
||||
}
|
||||
candidate.ID = domain.NewID()
|
||||
candidate.Detail = o.Message
|
||||
candidate.FirstSeen = firstOr(o.First, at)
|
||||
candidate.LastSeen = firstOr(o.Last, at)
|
||||
e, err := t.append(domain.EventObservationIncidentOpened, candidate)
|
||||
if err != nil {
|
||||
return appended, err
|
||||
}
|
||||
appended = append(appended, e)
|
||||
t.counts[candidate.ID] = o.Count
|
||||
t.seen[r.WorkerID+"\x00"+signature] = watermark{last: candidate.LastSeen, count: o.Count}
|
||||
continue
|
||||
}
|
||||
// Open already: accumulate, append nothing. A count lower than the last
|
||||
// one means the ring evicted the entry and started it again.
|
||||
delta := o.Count - t.counts[existing.ID]
|
||||
if delta < 0 {
|
||||
delta = o.Count
|
||||
}
|
||||
t.counts[existing.ID] += delta
|
||||
if last := firstOr(o.Last, at); last.After(existing.LastSeen) {
|
||||
existing.LastSeen = last
|
||||
t.Store.NoteObservation(existing) // last_seen is durable at close
|
||||
}
|
||||
t.seen[r.WorkerID+"\x00"+signature] = watermark{last: existing.LastSeen, count: o.Count}
|
||||
}
|
||||
|
||||
// The boundaries. A lease that ended, an epoch that changed, and a
|
||||
// worker-level incident whose last occurrence has gone stale.
|
||||
closed, err := t.closeWhere(at, "", func(inc domain.ObservationIncident) bool {
|
||||
if inc.WorkerID != r.WorkerID {
|
||||
return false
|
||||
}
|
||||
// Only an incident bound to a live lease has a lease boundary to close
|
||||
// it. One read out of the ring about a task that already finished has
|
||||
// no such boundary, so it ends the way a worker-level incident does.
|
||||
if inc.LeaseEpoch == "" {
|
||||
return at.Sub(inc.LastSeen) > QuietTimeout
|
||||
}
|
||||
return inc.TaskID != r.TaskID || inc.LeaseEpoch != r.LeaseEpoch
|
||||
})
|
||||
appended = append(appended, closed...)
|
||||
return appended, err
|
||||
}
|
||||
|
||||
// closeWhere finalizes every open incident the predicate selects. A reason of
|
||||
// "" is resolved per incident, which is what lets one sweep close a lease that
|
||||
// ended and a worker-level incident that went quiet.
|
||||
func (t *ObservationTracker) closeWhere(at time.Time, reason domain.ObservationCloseReason, match func(domain.ObservationIncident) bool) ([]domain.Event, error) {
|
||||
var out []domain.Event
|
||||
for _, inc := range t.Store.OpenObservations() {
|
||||
if !match(inc) {
|
||||
continue
|
||||
}
|
||||
inc.ClosedAt = at
|
||||
inc.RepeatCount = t.counts[inc.ID]
|
||||
inc.CloseReason = reason
|
||||
if inc.CloseReason == "" {
|
||||
switch {
|
||||
case inc.LeaseEpoch == "":
|
||||
inc.CloseReason = domain.ObservationCloseQuietTimeout
|
||||
case inc.TaskID != "":
|
||||
inc.CloseReason = domain.ObservationCloseEpochChange
|
||||
default:
|
||||
inc.CloseReason = domain.ObservationCloseLeaseEnd
|
||||
}
|
||||
}
|
||||
e, err := t.append(domain.EventObservationIncidentClosed, inc)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
delete(t.counts, inc.ID)
|
||||
out = append(out, e)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// append writes the incident as a worker-scoped event. The task it happened
|
||||
// during is carried in the payload rather than in Event.TaskID on purpose: an
|
||||
// incident is evidence about a worker, and binding it to the task aggregate
|
||||
// would bump that task's version from a path the lease knows nothing about.
|
||||
func (t *ObservationTracker) append(typ string, inc domain.ObservationIncident) (domain.Event, error) {
|
||||
b, err := json.Marshal(inc)
|
||||
if err != nil {
|
||||
return domain.Event{}, err
|
||||
}
|
||||
// "system" is the same aggregate QuotaReported uses for worker-scoped
|
||||
// facts: every event needs a task id, and this evidence belongs to a
|
||||
// worker rather than to any one task.
|
||||
e := domain.Event{ID: domain.NewID(), Type: typ, TaskID: "system", Payload: b, Surface: string(authz.System)}
|
||||
if err := t.Store.Append(e); err != nil {
|
||||
return domain.Event{}, fmt.Errorf("record observation incident: %w", err)
|
||||
}
|
||||
return e, nil
|
||||
}
|
||||
|
||||
func firstOr(t, fallback time.Time) time.Time {
|
||||
if t.IsZero() {
|
||||
return fallback
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// RecordIntervention writes down a repair the operator made by hand. It is
|
||||
// deliberately an explicit act: Orchestra cannot see a worker someone
|
||||
// restarted or a transaction someone deleted, and inferring "an operator
|
||||
// probably intervened" from a gap in the log would put guesses into the
|
||||
// evidence the ledger is built from.
|
||||
func RecordIntervention(s *store.Store, surface authz.Surface, in domain.OperatorIntervention) (domain.Event, error) {
|
||||
if err := in.Validate(); err != nil {
|
||||
return domain.Event{}, err
|
||||
}
|
||||
if in.At.IsZero() {
|
||||
in.At = time.Now().UTC()
|
||||
}
|
||||
taskID := in.TaskID
|
||||
version := 0
|
||||
if taskID != "" {
|
||||
t, ok := s.Task(taskID)
|
||||
if !ok {
|
||||
return domain.Event{}, domain.ErrNotFound
|
||||
}
|
||||
version = t.Version + 1
|
||||
} else {
|
||||
// A repair with no task is still about this deployment, so it lands on
|
||||
// the same aggregate the other worker-scoped facts use.
|
||||
taskID = "system"
|
||||
}
|
||||
b, err := json.Marshal(in)
|
||||
if err != nil {
|
||||
return domain.Event{}, err
|
||||
}
|
||||
e := domain.Event{
|
||||
ID: domain.NewID(), Type: domain.EventOperatorInterventionRecorded,
|
||||
TaskID: taskID, Version: version, At: in.At, Payload: b, Surface: string(surface),
|
||||
}
|
||||
if err := s.Append(e); err != nil {
|
||||
return domain.Event{}, err
|
||||
}
|
||||
return e, nil
|
||||
}
|
||||
@@ -0,0 +1,295 @@
|
||||
package operations
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"orchestra/internal/domain"
|
||||
"orchestra/internal/store"
|
||||
)
|
||||
|
||||
func tracker(t *testing.T) (*ObservationTracker, *store.Store) {
|
||||
t.Helper()
|
||||
s, err := store.Open(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return &ObservationTracker{Store: s}, s
|
||||
}
|
||||
|
||||
func ring(message string, count int, last time.Time) []domain.WorkerObservation {
|
||||
return []domain.WorkerObservation{{Message: message, Count: count, First: last.Add(-time.Minute), Last: last}}
|
||||
}
|
||||
|
||||
func closedIncident(t *testing.T, s *store.Store) domain.ObservationIncident {
|
||||
t.Helper()
|
||||
var out domain.ObservationIncident
|
||||
found := 0
|
||||
for _, e := range s.Events(0) {
|
||||
if e.Type != domain.EventObservationIncidentClosed {
|
||||
continue
|
||||
}
|
||||
found++
|
||||
if err := json.Unmarshal(e.Payload, &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if found != 1 {
|
||||
t.Fatalf("closed incidents = %d, want 1", found)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// 301 repeats of one refusal are one incident with an intensity of 301, not
|
||||
// 301 pieces of evidence. Appending each would spam the log and make one stuck
|
||||
// loop look like chronic, recurring debt.
|
||||
func TestRepeatsAreOneIncident(t *testing.T) {
|
||||
tr, s := tracker(t)
|
||||
at := time.Unix(1700000000, 0).UTC()
|
||||
report := func(count int, when time.Time) {
|
||||
if _, err := tr.Ingest(WorkerReport{
|
||||
WorkerID: "workpc-claude", Incarnation: "boot-1", TaskID: "task-a", LeaseEpoch: "epoch-1",
|
||||
Observations: ring("release task-a commit: 409 superseded", count, when), At: when,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
report(1, at)
|
||||
report(40, at.Add(time.Minute))
|
||||
report(301, at.Add(2*time.Minute))
|
||||
|
||||
opened := 0
|
||||
for _, e := range s.Events(0) {
|
||||
if e.Type == domain.EventObservationIncidentOpened {
|
||||
opened++
|
||||
}
|
||||
if e.Type == domain.EventObservationIncidentClosed {
|
||||
t.Fatal("an incident was closed while its lease was still running")
|
||||
}
|
||||
}
|
||||
if opened != 1 {
|
||||
t.Fatalf("opened %d incidents for one repeating failure", opened)
|
||||
}
|
||||
if open := s.OpenObservations(); len(open) != 1 || open[0].TaskID != "task-a" {
|
||||
t.Fatalf("open incidents = %+v", open)
|
||||
}
|
||||
}
|
||||
|
||||
// The ring is a bounded history, so an entry that disappears may have been
|
||||
// evicted rather than resolved. Absence must not close anything, and a
|
||||
// recreated entry must accumulate rather than restart its count.
|
||||
func TestEvictionNeitherClosesNorRestartsTheCount(t *testing.T) {
|
||||
tr, s := tracker(t)
|
||||
at := time.Unix(1700000000, 0).UTC()
|
||||
send := func(obs []domain.WorkerObservation, when time.Time) {
|
||||
if _, err := tr.Ingest(WorkerReport{
|
||||
WorkerID: "w", Incarnation: "boot-1", TaskID: "task-a", LeaseEpoch: "epoch-1",
|
||||
Observations: obs, At: when,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
send(ring("lease task-a not renewed: agent idle", 34, at), at)
|
||||
// Evicted: the message is simply gone from this heartbeat.
|
||||
send(nil, at.Add(time.Minute))
|
||||
if len(s.OpenObservations()) != 1 {
|
||||
t.Fatal("an incident was closed because its message left a bounded ring")
|
||||
}
|
||||
// Recreated, counting from scratch on the worker side.
|
||||
send(ring("lease task-a not renewed: agent idle", 3, at.Add(2*time.Minute)), at.Add(2*time.Minute))
|
||||
|
||||
// The lease ends, which is a real boundary.
|
||||
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "boot-1", At: at.Add(3 * time.Minute)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inc := closedIncident(t, s)
|
||||
if inc.RepeatCount != 37 {
|
||||
t.Fatalf("repeat_count = %d, want 37 (34 before eviction plus 3 after)", inc.RepeatCount)
|
||||
}
|
||||
if inc.CloseReason != domain.ObservationCloseEpochChange {
|
||||
t.Fatalf("close_reason = %q", inc.CloseReason)
|
||||
}
|
||||
if !inc.LastSeen.Equal(at.Add(2 * time.Minute)) {
|
||||
t.Fatalf("last_seen = %s, want the last actual occurrence", inc.LastSeen)
|
||||
}
|
||||
if !inc.ClosedAt.After(inc.LastSeen) {
|
||||
t.Fatal("closed_at must be when Orchestra finalized it, not when the failure last happened")
|
||||
}
|
||||
}
|
||||
|
||||
// Recurrence is independent incidents. The same signature on two tasks is two,
|
||||
// which is the evidence that means something; repeats inside one are intensity.
|
||||
func TestTheSameSignatureOnAnotherTaskIsASecondIncident(t *testing.T) {
|
||||
tr, s := tracker(t)
|
||||
at := time.Unix(1700000000, 0).UTC()
|
||||
if _, err := tr.Ingest(WorkerReport{
|
||||
WorkerID: "w", Incarnation: "boot-1", TaskID: "task-a", LeaseEpoch: "e1",
|
||||
Observations: ring("lease task-a not renewed: agent idle", 5, at), At: at,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tr.Ingest(WorkerReport{
|
||||
WorkerID: "w", Incarnation: "boot-1", TaskID: "task-b", LeaseEpoch: "e2",
|
||||
Observations: ring("lease task-b not renewed: agent idle", 2, at.Add(time.Minute)), At: at.Add(time.Minute),
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
opened, closed := 0, 0
|
||||
for _, e := range s.Events(0) {
|
||||
switch e.Type {
|
||||
case domain.EventObservationIncidentOpened:
|
||||
opened++
|
||||
case domain.EventObservationIncidentClosed:
|
||||
closed++
|
||||
}
|
||||
}
|
||||
if opened != 2 {
|
||||
t.Fatalf("opened = %d, want one incident per lease", opened)
|
||||
}
|
||||
if closed != 1 {
|
||||
t.Fatalf("closed = %d, want the first lease finalized when the second began", closed)
|
||||
}
|
||||
}
|
||||
|
||||
// A restart cannot continue the previous process's symptom.
|
||||
func TestAWorkerRestartClosesItsIncidents(t *testing.T) {
|
||||
tr, s := tracker(t)
|
||||
at := time.Unix(1700000000, 0).UTC()
|
||||
if _, err := tr.Ingest(WorkerReport{
|
||||
WorkerID: "w", Incarnation: "boot-1", TaskID: "task-a", LeaseEpoch: "e1",
|
||||
Observations: ring("herdr unreachable", 9, at), At: at,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "boot-2", At: at.Add(time.Minute)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inc := closedIncident(t, s)
|
||||
if inc.CloseReason != domain.ObservationCloseWorkerRestart || inc.RepeatCount != 9 {
|
||||
t.Fatalf("incident = %+v", inc)
|
||||
}
|
||||
}
|
||||
|
||||
// An observation with no lease has no terminal boundary, so staleness of its
|
||||
// last actual occurrence is what ends it.
|
||||
func TestAWorkerLevelIncidentClosesOnQuietTimeout(t *testing.T) {
|
||||
tr, s := tracker(t)
|
||||
at := time.Unix(1700000000, 0).UTC()
|
||||
if _, err := tr.Ingest(WorkerReport{
|
||||
WorkerID: "w", Incarnation: "boot-1",
|
||||
Observations: ring("heartbeat: connection refused", 4, at), At: at,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "boot-1", At: at.Add(time.Minute)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(s.OpenObservations()) != 1 {
|
||||
t.Fatal("a worker-level incident closed before its quiet timeout")
|
||||
}
|
||||
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "boot-1", At: at.Add(QuietTimeout + time.Minute)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if inc := closedIncident(t, s); inc.CloseReason != domain.ObservationCloseQuietTimeout {
|
||||
t.Fatalf("close_reason = %q", inc.CloseReason)
|
||||
}
|
||||
}
|
||||
|
||||
// The signature is what makes recurrence countable across tasks.
|
||||
func TestSignatureCollapsesIdsAndCounts(t *testing.T) {
|
||||
a := domain.ObservationSignature("lease 06G4WJ9T4F35NZC4Z8QQXM9Z6G not renewed: agent status idle and pane unchanged")
|
||||
b := domain.ObservationSignature("lease 06G4VF5HZW7Q4JBM3TTY7W1Y64 not renewed: agent status idle and pane unchanged")
|
||||
if a != b {
|
||||
t.Fatalf("the same failure on two tasks has two signatures:\n%s\n%s", a, b)
|
||||
}
|
||||
if c := domain.ObservationSignature("release 06G4WJ9T4F35NZC4Z8QQXM9Z6G commit: 409 superseded"); c == a {
|
||||
t.Fatal("two different failures collapsed to one signature")
|
||||
}
|
||||
}
|
||||
|
||||
// The first live run of the ledger caught this: a pane name carries the task
|
||||
// id in lower case, so the same failure signed differently on every task and
|
||||
// could never accumulate recurrence.
|
||||
func TestSignatureCollapsesAPaneName(t *testing.T) {
|
||||
a := domain.ObservationSignature(`phase rotation 06G4XAFH1MBPC35VSJN7V3NS14: pane orchestra-06g4xafh1mbpc35vsjn7v3ns14-be13b045:1.0 still holds input`)
|
||||
b := domain.ObservationSignature(`phase rotation 06G4WW6TND26M16CZA6WE5T458: pane orchestra-06g4ww6tnd26m16cza6we5t458-4d839c05:1.0 still holds input`)
|
||||
if a != b {
|
||||
t.Fatalf("one failure has two signatures:\n%s\n%s", a, b)
|
||||
}
|
||||
}
|
||||
|
||||
// The first live run manufactured recurrence out of one old failure: a quiet
|
||||
// timeout closed the incident, the entry stayed in the ring because the ring
|
||||
// is a bounded history rather than a set of live conditions, and every later
|
||||
// heartbeat opened it again. Four incidents, one failure, no new occurrence.
|
||||
func TestAClosedIncidentDoesNotReopenFromAStaleRingEntry(t *testing.T) {
|
||||
tr, s := tracker(t)
|
||||
at := time.Unix(1700000000, 0).UTC()
|
||||
entry := ring("heartbeat: connection refused", 4, at)
|
||||
if _, err := tr.Ingest(WorkerReport{WorkerID: "w", Incarnation: "b1", Observations: entry, At: at}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Long enough to close on quiet timeout, with the entry still reported.
|
||||
quiet := at.Add(QuietTimeout + time.Minute)
|
||||
for i := 0; i < 4; i++ {
|
||||
if _, err := tr.Ingest(WorkerReport{
|
||||
WorkerID: "w", Incarnation: "b1", Observations: entry,
|
||||
At: quiet.Add(time.Duration(i) * time.Minute),
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
opened, closed := 0, 0
|
||||
for _, e := range s.Events(0) {
|
||||
switch e.Type {
|
||||
case domain.EventObservationIncidentOpened:
|
||||
opened++
|
||||
case domain.EventObservationIncidentClosed:
|
||||
closed++
|
||||
}
|
||||
}
|
||||
if opened != 1 || closed != 1 {
|
||||
t.Fatalf("opened=%d closed=%d for one failure that never happened again", opened, closed)
|
||||
}
|
||||
|
||||
// A real new occurrence, which the ring shows by advancing the entry.
|
||||
later := quiet.Add(time.Hour)
|
||||
if _, err := tr.Ingest(WorkerReport{
|
||||
WorkerID: "w", Incarnation: "b1", Observations: ring("heartbeat: connection refused", 5, later), At: later,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reopened := 0
|
||||
for _, e := range s.Events(0) {
|
||||
if e.Type == domain.EventObservationIncidentOpened {
|
||||
reopened++
|
||||
}
|
||||
}
|
||||
if reopened != 2 {
|
||||
t.Fatalf("a genuine new occurrence did not open an incident: opened=%d", reopened)
|
||||
}
|
||||
}
|
||||
|
||||
// The ring outlives the work it describes, so the task comes from the message
|
||||
// rather than from whatever the worker happens to be running now.
|
||||
func TestTheTaskComesFromTheMessageNotTheCurrentLease(t *testing.T) {
|
||||
tr, s := tracker(t)
|
||||
at := time.Unix(1700000000, 0).UTC()
|
||||
if _, err := tr.Ingest(WorkerReport{
|
||||
WorkerID: "w", Incarnation: "b1", TaskID: "06G4XAFH1MBPC35VSJN7V3NS14", LeaseEpoch: "now",
|
||||
Observations: ring("renew lease 06G4WW6TND26M16CZA6WE5T458: 409 conflict", 3, at), At: at,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
open := s.OpenObservations()
|
||||
if len(open) != 1 {
|
||||
t.Fatalf("open = %+v", open)
|
||||
}
|
||||
if open[0].TaskID != "06G4WW6TND26M16CZA6WE5T458" {
|
||||
t.Fatalf("the failure was attributed to the wrong task: %q", open[0].TaskID)
|
||||
}
|
||||
if open[0].LeaseEpoch != "" {
|
||||
t.Fatalf("an old failure inherited the current lease's epoch: %q", open[0].LeaseEpoch)
|
||||
}
|
||||
}
|
||||
@@ -154,11 +154,13 @@ func blockForPlanMismatch(s *store.Store, taskID string, m domain.PlanMismatch)
|
||||
packet += "- evidence: " + oneLine(e) + "\n"
|
||||
}
|
||||
packet += "\nReply to say how to proceed. Your reply becomes a recorded decision and outranks the plan. If it resolves the contradiction, the task resumes on the same plan; say so explicitly if you want the plan rewritten instead.\n"
|
||||
b, err := json.Marshal(map[string]any{
|
||||
payload := map[string]any{
|
||||
"blocker": packet,
|
||||
"block_reason": string(domain.BlockReasonPlanMismatch),
|
||||
"lifecycle_phase": "awaiting_human",
|
||||
})
|
||||
}
|
||||
fenceToLease(payload, t)
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -108,6 +108,11 @@ func TestReplanKeepsTheOldPlanUntilAReplacementIsSealed(t *testing.T) {
|
||||
if len(during.PlanHistory) != 0 {
|
||||
t.Fatalf("the plan was moved to history early: %v", during.PlanHistory)
|
||||
}
|
||||
// F66: the planning session this reopen convenes has to be told what it is
|
||||
// there to fix, and the implementer that found it is gone by then.
|
||||
if during.PlanMismatch == nil || during.PlanMismatch.PhaseID != "phase-1" {
|
||||
t.Fatalf("the contradiction did not survive to the phase it reopened: %+v", during.PlanMismatch)
|
||||
}
|
||||
}
|
||||
|
||||
// Sealing the replacement is the moment the old plan is superseded. Progress
|
||||
@@ -151,6 +156,11 @@ func TestSealingTheReplacementSupersedesThePlanAndItsProgress(t *testing.T) {
|
||||
if _, err := s.Artifact(oldRef); err != nil {
|
||||
t.Fatalf("the superseded plan is unreadable: %v", err)
|
||||
}
|
||||
// The replacement answers the contradiction, so it stops being a live
|
||||
// instruction and stays in the log as history.
|
||||
if after, _ := s.Task(id); after.PlanMismatch != nil {
|
||||
t.Fatalf("a settled contradiction is still live: %+v", after.PlanMismatch)
|
||||
}
|
||||
assertPhase(t, s, id, domain.WorkPhaseImplement)
|
||||
}
|
||||
|
||||
@@ -215,7 +225,25 @@ func TestHumanAnswerResumesTheSamePlanWithoutResealing(t *testing.T) {
|
||||
if !PlanMismatchAnswered(s, id) {
|
||||
t.Fatal("the human answered and the task is still waiting")
|
||||
}
|
||||
// F64: asserting the predicate is not asserting the resume. This test
|
||||
// passed for as long as the predicate had no caller, while a task blocked
|
||||
// on a mismatch stayed blocked forever however the human replied.
|
||||
if events, err := ResumeAnsweredBlockers(s); err != nil || len(events) != 1 {
|
||||
t.Fatalf("an answered mismatch did not return to the queue: events=%v err=%v", events, err)
|
||||
}
|
||||
after, _ := s.Task(id)
|
||||
if after.State != domain.StateQueued {
|
||||
t.Fatalf("state = %s after the human answered, want queued", after.State)
|
||||
}
|
||||
// F67: the stop is over, so the projection must not keep reporting it.
|
||||
if after.BlockReason != "" || after.Blocker != "" {
|
||||
t.Fatalf("a resumed task still reports its blocker: %q %q", after.BlockReason, after.Blocker)
|
||||
}
|
||||
// The answer outranks the plan and stands as an ordinary decision, so the
|
||||
// contradiction it settled is not carried into later sessions either.
|
||||
if after.PlanMismatch != nil {
|
||||
t.Fatalf("an answered contradiction is still live: %+v", after.PlanMismatch)
|
||||
}
|
||||
if after.PlanRef != planRef {
|
||||
t.Fatal("answering the question replaced the plan")
|
||||
}
|
||||
|
||||
@@ -43,6 +43,13 @@ func PlanPhaseCommands(s *store.Store, project registry.Project, taskID, phaseID
|
||||
if t.PlanRef == "" {
|
||||
return workphase.PlanPhase{}, fmt.Errorf("%w: this task has no accepted plan", ErrPlanPhase)
|
||||
}
|
||||
// Verification is implementation work. A request that arrives in another
|
||||
// phase belongs to a trajectory Orchestra has already ended: run 20's
|
||||
// reopened planning session executed the implementer's leftover request
|
||||
// and recorded a phase of a plan that was being replaced.
|
||||
if current(t) != domain.WorkPhaseImplement {
|
||||
return workphase.PlanPhase{}, fmt.Errorf("%w: phase verification belongs to the implement phase, and this task is in %s", ErrPlanPhase, current(t))
|
||||
}
|
||||
raw, err := s.Artifact(t.PlanRef)
|
||||
if err != nil {
|
||||
return workphase.PlanPhase{}, fmt.Errorf("read accepted plan: %w", err)
|
||||
@@ -124,7 +131,12 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
|
||||
// was established.
|
||||
record.Status = domain.PlanPhaseAwaitingManual
|
||||
}
|
||||
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID) {
|
||||
// Carry the confirmed tree forward as provenance. Without it a second
|
||||
// rerun would compare against nothing and re-inherit the sign-off.
|
||||
if prior, ok := t.PlanPhase(phaseID); ok {
|
||||
record.ManualAtSHA = prior.ManualAtSHA
|
||||
}
|
||||
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID, record) {
|
||||
record.Status = domain.PlanPhaseVerified
|
||||
}
|
||||
if ref, err := s.PutArtifact(verificationEvidence(runs)); err == nil {
|
||||
@@ -135,7 +147,7 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
|
||||
payload := map[string]any{
|
||||
"plan_ref": record.PlanRef, "phase_id": record.PhaseID, "status": string(record.Status),
|
||||
"commands": record.Commands, "exit_codes": record.ExitCodes, "at_sha": record.AtSHA,
|
||||
"evidence_ref": record.EvidenceRef, "at": record.At,
|
||||
"evidence_ref": record.EvidenceRef, "at": record.At, "manual_at_sha": record.ManualAtSHA,
|
||||
}
|
||||
if t.Lease != nil {
|
||||
payload["harness_id"], payload["lease_epoch"] = t.Lease.HarnessID, t.Lease.Epoch
|
||||
@@ -149,9 +161,20 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
|
||||
}
|
||||
|
||||
// manuallySignedOff reports whether a human has already approved this exact
|
||||
// phase of this exact plan. The subject carries both, so a later "looks good"
|
||||
// on an unrelated thread cannot satisfy a gate nobody was discussing.
|
||||
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string) bool {
|
||||
// phase of this exact plan, against the tree this run examined. The subject
|
||||
// carries plan and phase, so a later "looks good" on an unrelated thread
|
||||
// cannot satisfy a gate nobody was discussing.
|
||||
//
|
||||
// The tree matters as much as the subject (F63). A sign-off is a human saying
|
||||
// they read what this code prints; an edit afterwards can change exactly that.
|
||||
// A record whose ManualAtSHA names a different commit is therefore not signed
|
||||
// off, and waits for the human again. A sign-off given before any run has no
|
||||
// confirmed tree to compare against and still counts, which keeps the ordinary
|
||||
// ordering unchanged.
|
||||
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string, record domain.PlanPhaseRecord) bool {
|
||||
if record.ManualAtSHA != "" && record.ManualAtSHA != record.AtSHA {
|
||||
return false
|
||||
}
|
||||
intent, err := s.EffectiveIntent(t.ID)
|
||||
if err != nil {
|
||||
return false
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
|
||||
const shaOne = "1111111111111111111111111111111111111111"
|
||||
const shaTwo = "2222222222222222222222222222222222222222"
|
||||
const shaThree = "3333333333333333333333333333333333333333"
|
||||
|
||||
func planProject() registry.Project {
|
||||
p := registry.Project{
|
||||
@@ -27,6 +28,11 @@ func planProject() registry.Project {
|
||||
func planWith(t *testing.T, markdown string) (*store.Store, registry.Project, string) {
|
||||
t.Helper()
|
||||
s, id := phaseStore(t)
|
||||
// Leased, because everything these tests drive comes from a live implement
|
||||
// session. Skipping it hid F65: two coordinator-side stops omitted the
|
||||
// fencing fields Store.Append requires on a leased task, and every test
|
||||
// passed because no test ever leased one.
|
||||
lease(t, s, id)
|
||||
project := planProject()
|
||||
if _, err := AdvanceWorkPhase(s, project, id, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -251,6 +257,14 @@ func TestLegacyPlanIsExplicitlyNonProgressable(t *testing.T) {
|
||||
// signOff records a human decision bound to one phase of one plan, which is
|
||||
// the only thing that satisfies a manual verification gate.
|
||||
func signOff(t *testing.T, s *store.Store, taskID, subject string) {
|
||||
t.Helper()
|
||||
signOffFrom(t, s, taskID, subject, "signoff-"+subject)
|
||||
}
|
||||
|
||||
// signOffFrom names the comment the sign-off came from. Two sign-offs on one
|
||||
// subject are a real sequence once a rerun sends a phase back to the human,
|
||||
// and provenance is unique per comment.
|
||||
func signOffFrom(t *testing.T, s *store.Store, taskID, subject, externalID string) {
|
||||
t.Helper()
|
||||
task, _ := s.Task(taskID)
|
||||
if err := s.Append(domain.Event{
|
||||
@@ -259,9 +273,87 @@ func signOff(t *testing.T, s *store.Store, taskID, subject string) {
|
||||
Payload: mustJSONBytes(t, map[string]any{
|
||||
"decision_id": domain.NewID(), "kind": "answer", "subject": subject,
|
||||
"value": "manual steps confirmed",
|
||||
"source": map[string]any{"provider": "gitea", "external_id": "signoff-" + subject},
|
||||
"source": map[string]any{"provider": "gitea", "external_id": externalID},
|
||||
}),
|
||||
}); err != nil {
|
||||
t.Fatalf("sign off: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// F63, found live on run 19. A manual sign-off says a human read what this
|
||||
// code prints. An edit afterwards can change exactly that, so rerunning the
|
||||
// automated half at a new commit must not carry the human half with it.
|
||||
func TestASignOffDoesNotSurviveTheTreeItWasGivenAgainst(t *testing.T) {
|
||||
s, project, id := planWith(t, twoPhasePlan)
|
||||
run := []VerificationRun{{Command: []string{"go", "test", "./internal/..."}, ExitCode: 0}}
|
||||
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaOne, run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ := s.Task(id)
|
||||
signOff(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"))
|
||||
task, _ = s.Task(id)
|
||||
rec, _ := task.PlanPhase("phase-2")
|
||||
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaOne {
|
||||
t.Fatalf("sign-off did not bind to the tree it read: %+v", rec)
|
||||
}
|
||||
|
||||
// The tree moves and the phase is re-verified. The commands pass again;
|
||||
// the human has not seen the new output.
|
||||
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaTwo, run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ = s.Task(id)
|
||||
rec, _ = task.PlanPhase("phase-2")
|
||||
if rec.Status != domain.PlanPhaseAwaitingManual {
|
||||
t.Fatalf("status = %q at a tree the human never saw, want awaiting_manual_verification", rec.Status)
|
||||
}
|
||||
if rec.ManualAtSHA != shaOne {
|
||||
t.Fatalf("the confirmed tree was lost: %+v", rec)
|
||||
}
|
||||
|
||||
// A second rerun must not re-inherit it either, which is what carrying
|
||||
// ManualAtSHA forward is for.
|
||||
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaThree, run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ = s.Task(id)
|
||||
rec, _ = task.PlanPhase("phase-2")
|
||||
if rec.Status != domain.PlanPhaseAwaitingManual {
|
||||
t.Fatalf("a second rerun re-inherited the sign-off: %q", rec.Status)
|
||||
}
|
||||
|
||||
// Signing off again, on the tree that is now current, verifies it.
|
||||
signOffFrom(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"), "signoff-second")
|
||||
task, _ = s.Task(id)
|
||||
rec, _ = task.PlanPhase("phase-2")
|
||||
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaThree {
|
||||
t.Fatalf("a fresh sign-off did not verify the current tree: %+v", rec)
|
||||
}
|
||||
}
|
||||
|
||||
// Run 20: a replan reopened the plan phase, the implementer's leftover
|
||||
// verification request outlived its session, and the planning session that
|
||||
// replaced it executed the request. Orchestra recorded a verified phase of the
|
||||
// plan it was in the middle of replacing.
|
||||
func TestVerificationIsRefusedOutsideImplement(t *testing.T) {
|
||||
s, project, id := planWith(t, twoPhasePlan)
|
||||
task, _ := s.Task(id)
|
||||
m := mismatch(task.PlanRef)
|
||||
m.RequestedAction = domain.PlanMismatchReplan
|
||||
if _, err := RecordPlanMismatch(s, project, id, m, shaOne); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assertPhase(t, s, id, domain.WorkPhasePlan)
|
||||
|
||||
_, err := RecordPlanPhaseVerification(s, project, id, "phase-1", shaOne,
|
||||
[]VerificationRun{{Command: []string{"go", "build", "./..."}, ExitCode: 0}})
|
||||
if !errors.Is(err, ErrPlanPhase) {
|
||||
t.Fatalf("a reopened task verified a phase of the plan being replaced: %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "implement") {
|
||||
t.Fatalf("the refusal does not say which phase owns verification: %v", err)
|
||||
}
|
||||
if after, _ := s.Task(id); len(after.PlanPhases()) != 0 {
|
||||
t.Fatalf("progress was recorded anyway: %+v", after.PlanPhases())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -67,11 +67,13 @@ func raiseTrajectoryGate(s *store.Store, t domain.Task, from, to domain.WorkPhas
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := json.Marshal(map[string]any{
|
||||
payload := map[string]any{
|
||||
"blocker": packet,
|
||||
"block_reason": string(domain.BlockReasonTrajectoryGate),
|
||||
"lifecycle_phase": "awaiting_human",
|
||||
})
|
||||
}
|
||||
fenceToLease(payload, t)
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -102,6 +102,36 @@ func projectDebt(events []domain.Event, readArtifact func(string) ([]byte, error
|
||||
o.Detail = str(p["last_error"])
|
||||
add(class, domain.DebtSignature(class, failure, harness, "lease"), o,
|
||||
"tasks end in "+failure, review.Important)
|
||||
case domain.EventObservationIncidentClosed:
|
||||
// One incident, whatever it repeated. The debt class comes from
|
||||
// the signature's shape rather than a failure class, because a
|
||||
// worker observation is a symptom the worker described, not a
|
||||
// lifecycle outcome Orchestra decided.
|
||||
var inc domain.ObservationIncident
|
||||
if json.Unmarshal(e.Payload, &inc) != nil || inc.Signature == "" {
|
||||
continue
|
||||
}
|
||||
o := base
|
||||
o.Kind = domain.ObservationWorkerFailure
|
||||
o.TaskID = inc.TaskID
|
||||
o.Detail = inc.Detail
|
||||
o.Repeats = inc.RepeatCount
|
||||
add(domain.DebtOperational,
|
||||
domain.DebtSignature(domain.DebtOperational, inc.Signature, inc.WorkerID, "worker"),
|
||||
o, "workers report "+inc.Signature, review.Important)
|
||||
case domain.EventOperatorInterventionRecorded:
|
||||
var in domain.OperatorIntervention
|
||||
if json.Unmarshal(e.Payload, &in) != nil || !in.Kind.Valid() {
|
||||
continue
|
||||
}
|
||||
o := base
|
||||
o.Kind = domain.ObservationManualIntervention
|
||||
o.TaskID = in.TaskID
|
||||
o.Detail = in.Reason
|
||||
o.Paths = in.Components
|
||||
add(domain.DebtOperational,
|
||||
domain.DebtSignature(domain.DebtOperational, string(in.Kind), in.WorkerID, "manual"),
|
||||
o, "an operator repairs this by hand ("+string(in.Kind)+")", review.Important)
|
||||
case domain.EventPlanMismatchRecorded:
|
||||
o := base
|
||||
o.Kind = domain.ObservationPlanMismatch
|
||||
@@ -167,16 +197,15 @@ func projectDebt(events []domain.Event, readArtifact func(string) ([]byte, error
|
||||
// hole in the system. A kind the log could carry and does not is a fact about
|
||||
// this history.
|
||||
func debtGaps(seen map[domain.ObservationKind]bool) []domain.EvidenceGap {
|
||||
gaps := []domain.EvidenceGap{
|
||||
{Kind: domain.ObservationManualIntervention, Durable: false,
|
||||
Reason: "no event type records an operator repair, so every manual recovery is invisible to this ledger"},
|
||||
{Kind: domain.ObservationWorkerFailure, Durable: false,
|
||||
Reason: "worker observations live in worker memory and reach the coordinator only inside WorkerHealth, which is not persisted"},
|
||||
}
|
||||
// Both of these were once permanent holes in the system. They are ordinary
|
||||
// evidence now, so their absence is a fact about this history rather than
|
||||
// about Orchestra.
|
||||
var gaps []domain.EvidenceGap
|
||||
for _, k := range []domain.ObservationKind{
|
||||
domain.ObservationBlockReason, domain.ObservationFailureClass,
|
||||
domain.ObservationReviewFinding, domain.ObservationPlanMismatch,
|
||||
domain.ObservationDeferredFinding,
|
||||
domain.ObservationDeferredFinding, domain.ObservationWorkerFailure,
|
||||
domain.ObservationManualIntervention,
|
||||
} {
|
||||
if !seen[k] {
|
||||
gaps = append(gaps, domain.EvidenceGap{Kind: k, Durable: true,
|
||||
|
||||
@@ -64,24 +64,92 @@ func TestProjectDebtIgnoresOrdinaryLifecycleStops(t *testing.T) {
|
||||
// not recorded anywhere".
|
||||
func TestProjectDebtReportsWhatItCannotSee(t *testing.T) {
|
||||
ledger := ProjectDebt(nil)
|
||||
var manual, worker bool
|
||||
for _, g := range ledger.Gaps {
|
||||
if g.Durable {
|
||||
continue
|
||||
}
|
||||
switch g.Kind {
|
||||
case domain.ObservationManualIntervention:
|
||||
manual = true
|
||||
case domain.ObservationWorkerFailure:
|
||||
worker = true
|
||||
}
|
||||
}
|
||||
if !manual || !worker {
|
||||
t.Fatalf("the two known holes must always be reported: %+v", ledger.Gaps)
|
||||
}
|
||||
for _, g := range ledger.Gaps {
|
||||
if g.Reason == "" {
|
||||
t.Fatalf("gap %q has no reason", g.Kind)
|
||||
}
|
||||
// Slice B closed the two holes this ledger used to report about
|
||||
// itself. Every silence is now a fact about one history, never a kind
|
||||
// of evidence the system cannot record at all.
|
||||
if !g.Durable {
|
||||
t.Fatalf("gap %q is reported as unrecordable: %+v", g.Kind, g)
|
||||
}
|
||||
}
|
||||
var worker, manual bool
|
||||
for _, g := range ledger.Gaps {
|
||||
switch g.Kind {
|
||||
case domain.ObservationWorkerFailure:
|
||||
worker = true
|
||||
case domain.ObservationManualIntervention:
|
||||
manual = true
|
||||
}
|
||||
}
|
||||
if !worker || !manual {
|
||||
t.Fatalf("an empty history should still name both kinds as absent: %+v", ledger.Gaps)
|
||||
}
|
||||
}
|
||||
|
||||
// The whole point of incidents. One worker stuck in a retry loop must not
|
||||
// manufacture recurrence, while its intensity is still on the record.
|
||||
func TestRecurrenceCountsIncidentsAndKeepsIntensitySeparate(t *testing.T) {
|
||||
closed := func(id, worker, task, epoch string, repeats int) domain.Event {
|
||||
b, _ := json.Marshal(domain.ObservationIncident{
|
||||
ID: id, WorkerID: worker, TaskID: task, LeaseEpoch: epoch,
|
||||
Signature: "lease <id> not renewed: agent status idle and pane unchanged",
|
||||
Detail: "lease " + task + " not renewed: agent status idle and pane unchanged",
|
||||
RepeatCount: repeats, CloseReason: domain.ObservationCloseEpochChange,
|
||||
})
|
||||
return domain.Event{ID: id, Type: domain.EventObservationIncidentClosed, TaskID: "system", Payload: b}
|
||||
}
|
||||
ledger := ProjectDebt([]domain.Event{
|
||||
closed("i1", "workpc-claude", "task-a", "e1", 301),
|
||||
closed("i2", "workpc-claude", "task-b", "e2", 2),
|
||||
})
|
||||
if len(ledger.Items) != 1 {
|
||||
t.Fatalf("one kind of failure produced %d items", len(ledger.Items))
|
||||
}
|
||||
item := ledger.Items[0]
|
||||
if len(item.Observations) != 2 {
|
||||
t.Fatalf("recurrence = %d, want one per incident", len(item.Observations))
|
||||
}
|
||||
intensity := 0
|
||||
for _, o := range item.Observations {
|
||||
if o.Kind != domain.ObservationWorkerFailure {
|
||||
t.Fatalf("observation kind = %q", o.Kind)
|
||||
}
|
||||
intensity += o.Repeats
|
||||
}
|
||||
if intensity != 303 {
|
||||
t.Fatalf("intensity = %d, want 303 carried alongside a recurrence of 2", intensity)
|
||||
}
|
||||
tasks := map[string]bool{}
|
||||
for _, o := range item.Observations {
|
||||
tasks[o.TaskID] = true
|
||||
}
|
||||
if len(tasks) != 2 {
|
||||
t.Fatalf("the two incidents are not attributed to their tasks: %+v", item.Observations)
|
||||
}
|
||||
}
|
||||
|
||||
// A repair the operator made by hand is evidence like any other, once they say
|
||||
// it happened.
|
||||
func TestAnOperatorRepairBecomesDebtEvidence(t *testing.T) {
|
||||
b, _ := json.Marshal(domain.OperatorIntervention{
|
||||
WorkerID: "workpc-opencode", Kind: domain.InterventionTransactionClean,
|
||||
Reason: "deleted a release transaction stuck at prepared so the pane could be reused",
|
||||
})
|
||||
ledger := ProjectDebt([]domain.Event{{
|
||||
ID: "i1", Type: domain.EventOperatorInterventionRecorded, TaskID: "system", Payload: b,
|
||||
}})
|
||||
if len(ledger.Items) != 1 || len(ledger.Items[0].Observations) != 1 {
|
||||
t.Fatalf("the repair produced no debt evidence: %+v", ledger.Items)
|
||||
}
|
||||
if got := ledger.Items[0].Observations[0].Kind; got != domain.ObservationManualIntervention {
|
||||
t.Fatalf("kind = %q", got)
|
||||
}
|
||||
for _, g := range ledger.Gaps {
|
||||
if g.Kind == domain.ObservationManualIntervention {
|
||||
t.Fatal("manual intervention is still reported as missing from a history that contains one")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -49,6 +49,9 @@ type Store struct {
|
||||
cursors map[string]string
|
||||
cursorPath string
|
||||
decisionSource map[string]string
|
||||
// openObservations are the incidents opened and not yet closed, by id.
|
||||
// Derived from the log, so a restart finds them again.
|
||||
openObservations map[string]domain.ObservationIncident
|
||||
// PreLease runs immediately before a lease is minted, which is the single
|
||||
// point where ownership of a task begins. Reconciliation of newer human
|
||||
// input belongs here rather than in any individual launch path, because a
|
||||
@@ -96,7 +99,7 @@ func Open(dir string) (*Store, error) {
|
||||
if t, ok := s.tasks[e.TaskID]; ok && e.Version != t.Version+1 {
|
||||
return nil, domain.ErrConflict
|
||||
}
|
||||
global := e.Type == "QuotaReported" || e.Type == "StandupAdvisory" || e.Type == "ApprovalGranted" || e.Type == "ApprovalDenied"
|
||||
global := domain.EventWithoutTask(e.Type)
|
||||
if _, ok := s.tasks[e.TaskID]; !ok && e.Type != "TaskCreated" && !global {
|
||||
return nil, domain.ErrNotFound
|
||||
}
|
||||
@@ -272,6 +275,9 @@ func (s *Store) apply(e domain.Event) error {
|
||||
}
|
||||
if p.Subject == domain.PlanPhaseSubject(rec.PlanRef, rec.PhaseID) {
|
||||
t.PlanProgress.Phases[i].Status = domain.PlanPhaseVerified
|
||||
// Record which tree the sign-off was about, so a later run
|
||||
// at a different commit cannot inherit it (F63).
|
||||
t.PlanProgress.Phases[i].ManualAtSHA = rec.AtSHA
|
||||
t.Version = e.Version
|
||||
s.replaceTask(e.TaskID, t)
|
||||
}
|
||||
@@ -309,6 +315,9 @@ func (s *Store) apply(e domain.Event) error {
|
||||
}
|
||||
t.PlanRef = p.ArtifactRef
|
||||
t.PlanProgress = nil
|
||||
// The replacement answers the contradiction. Carrying it
|
||||
// further would put a solved problem in every later context.
|
||||
t.PlanMismatch = nil
|
||||
}
|
||||
}
|
||||
t.WorkPhase = p.Phase
|
||||
@@ -484,6 +493,39 @@ func (s *Store) apply(e domain.Event) error {
|
||||
t.Version = e.Version
|
||||
s.replaceTask(e.TaskID, t)
|
||||
return nil
|
||||
case domain.EventObservationIncidentOpened:
|
||||
// Open incidents are projected so a coordinator restart resumes them
|
||||
// instead of orphaning them half-recorded. The event is what makes an
|
||||
// incident durable at first sight; this is how it is found again.
|
||||
var inc domain.ObservationIncident
|
||||
if err := json.Unmarshal(e.Payload, &inc); err != nil {
|
||||
return err
|
||||
}
|
||||
if s.openObservations == nil {
|
||||
s.openObservations = map[string]domain.ObservationIncident{}
|
||||
}
|
||||
s.openObservations[inc.ID] = inc
|
||||
return nil
|
||||
case domain.EventObservationIncidentClosed:
|
||||
var inc domain.ObservationIncident
|
||||
if err := json.Unmarshal(e.Payload, &inc); err != nil {
|
||||
return err
|
||||
}
|
||||
delete(s.openObservations, inc.ID)
|
||||
return nil
|
||||
case domain.EventPlanMismatchRecorded:
|
||||
// Projected so the phase this reopens can be told what reopened it.
|
||||
// The event is the record; this is the live instruction derived from
|
||||
// it, and it stops being live as soon as a replacement seals or the
|
||||
// stop is answered.
|
||||
var m domain.PlanMismatch
|
||||
if err := json.Unmarshal(e.Payload, &m); err != nil {
|
||||
return err
|
||||
}
|
||||
t.PlanMismatch = &m
|
||||
t.Version = e.Version
|
||||
s.replaceTask(e.TaskID, t)
|
||||
return nil
|
||||
case domain.EventPlanPhaseVerified:
|
||||
var pp domain.PlanPhaseRecord
|
||||
if err := json.Unmarshal(e.Payload, &pp); err != nil {
|
||||
@@ -585,8 +627,22 @@ func (s *Store) apply(e domain.Event) error {
|
||||
// A question only stands while the task is blocked on it. Afterwards the
|
||||
// answer is an ordinary standing decision and the log still holds the
|
||||
// question, so keeping it on the task would put it in every later context.
|
||||
//
|
||||
// The blocker itself has the same lifetime and was outliving it (F67). A
|
||||
// task resumed from a plan mismatch kept reporting block_reason
|
||||
// plan_mismatch while it ran, so every surface reading the projection
|
||||
// showed a stop that had already been answered.
|
||||
if t.State != domain.StateBlocked {
|
||||
t.DecisionRequest = nil
|
||||
t.Blocker, t.BlockReason = "", ""
|
||||
// The contradiction is cleared only by the correction that answers the
|
||||
// stop, never by any other event that happens to find the task
|
||||
// unblocked. This block runs for every event, and clearing it here
|
||||
// unconditionally erased the contradiction at the very rotation the
|
||||
// reopen causes, which is exactly when the planner needs it.
|
||||
if e.Type == "TaskCorrected" {
|
||||
t.PlanMismatch = nil
|
||||
}
|
||||
}
|
||||
if phase, ok := p["lifecycle_phase"].(string); ok && phase != "" {
|
||||
t.LifecyclePhase = phase
|
||||
@@ -814,7 +870,7 @@ func (s *Store) Append(e domain.Event) error {
|
||||
return fmt.Errorf("%w: corrects references unknown event %q for this task", domain.ErrInvalid, corrects)
|
||||
}
|
||||
}
|
||||
global := e.Type == "QuotaReported" || e.Type == "StandupAdvisory" || e.Type == "ApprovalGranted" || e.Type == "ApprovalDenied"
|
||||
global := domain.EventWithoutTask(e.Type)
|
||||
if !taskExists && e.Type != "TaskCreated" && !global {
|
||||
return domain.ErrNotFound
|
||||
}
|
||||
@@ -876,7 +932,7 @@ func (s *Store) Append(e domain.Event) error {
|
||||
// for every lifecycle mutation.
|
||||
func (s *Store) validateTransition(e domain.Event, t domain.Task, exists bool, p map[string]any) error {
|
||||
if !exists {
|
||||
if e.Type != "TaskCreated" && e.Type != "QuotaReported" && e.Type != "StandupAdvisory" && e.Type != "ApprovalGranted" && e.Type != "ApprovalDenied" {
|
||||
if e.Type != "TaskCreated" && !domain.EventWithoutTask(e.Type) {
|
||||
return domain.ErrNotFound
|
||||
}
|
||||
return nil
|
||||
@@ -1080,6 +1136,31 @@ func (s *Store) Artifact(ref string) ([]byte, error) {
|
||||
return b, nil
|
||||
}
|
||||
|
||||
// NoteObservation updates an open incident's last actual occurrence. It writes
|
||||
// no event: the aggregate is durable when the incident is finalized, and
|
||||
// appending one per heartbeat is exactly the spam this design exists to avoid.
|
||||
func (s *Store) NoteObservation(inc domain.ObservationIncident) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if _, ok := s.openObservations[inc.ID]; ok {
|
||||
s.openObservations[inc.ID] = inc
|
||||
}
|
||||
}
|
||||
|
||||
// OpenObservations returns the incidents that are open, newest first by first
|
||||
// sight. Open means not yet finalized as durable evidence, never "the failure
|
||||
// is happening right now".
|
||||
func (s *Store) OpenObservations() []domain.ObservationIncident {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := make([]domain.ObservationIncident, 0, len(s.openObservations))
|
||||
for _, inc := range s.openObservations {
|
||||
out = append(out, inc)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].FirstSeen.After(out[j].FirstSeen) })
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *Store) Task(id string) (domain.Task, bool) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
@@ -723,3 +723,72 @@ func TestBlockRetainsLeaseEpochForALaterReopen(t *testing.T) {
|
||||
t.Fatalf("last lease epoch %q, want %q", after.LastLeaseEpoch, epoch)
|
||||
}
|
||||
}
|
||||
|
||||
// F66's projection has to survive the rotation the reopen causes. The session
|
||||
// that reported the contradiction hands off, a successor leases, and only then
|
||||
// is the planning context rendered. Live on run 21, the field was gone by
|
||||
// then: the reopen recorded it and the rotation lost it.
|
||||
func TestTheContradictionSurvivesTheRotationItCauses(t *testing.T) {
|
||||
s, err := Open(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b := []byte(`{"source":"s","external_id":"x","project":"p"}`)
|
||||
if err := s.Append(domain.Event{ID: "create", Type: "TaskCreated", TaskID: "t", Version: 1, Payload: b, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Lease("t", "h", time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Frame to implement, the shortest legal route to the phase a mismatch
|
||||
// can be reported from.
|
||||
task, _ := s.Task("t")
|
||||
toImplement, _ := json.Marshal(map[string]any{"phase": "implement", "from": "frame"})
|
||||
if err := s.Append(domain.Event{ID: "impl", Type: domain.EventWorkPhaseChanged, TaskID: "t", Version: task.Version + 1, Payload: toImplement, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ = s.Task("t")
|
||||
m, _ := json.Marshal(map[string]any{
|
||||
"plan_ref": "plan-a", "phase_id": "phase-2",
|
||||
"at_sha": "0123456789012345678901234567890123456789",
|
||||
"observed": "the body is assembled inline",
|
||||
"contradicts": "the plan says one helper returns it",
|
||||
"harness_id": task.Lease.HarnessID,
|
||||
"lease_epoch": task.Lease.Epoch,
|
||||
"requested_action": "replan",
|
||||
})
|
||||
if err := s.Append(domain.Event{ID: "mismatch", Type: domain.EventPlanMismatchRecorded, TaskID: "t", Version: task.Version + 1, Payload: m, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, _ := s.Task("t"); got.PlanMismatch == nil {
|
||||
t.Fatal("the contradiction was not projected at all")
|
||||
}
|
||||
|
||||
// The reopen, then the rotation it causes.
|
||||
task, _ = s.Task("t")
|
||||
ph, _ := json.Marshal(map[string]any{"phase": "plan", "from": "implement", "reopen": domain.EventPlanMismatchRecorded, "reopen_phase_id": "phase-2"})
|
||||
if err := s.Append(domain.Event{ID: "reopen", Type: domain.EventWorkPhaseChanged, TaskID: "t", Version: task.Version + 1, Payload: ph, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ref, err := s.PutArtifact([]byte("handoff"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ = s.Task("t")
|
||||
rel, _ := json.Marshal(map[string]any{"handoff_ref": ref, "anchor_sha": "0123456789012345678901234567890123456789", "harness_id": task.Lease.HarnessID, "lease_epoch": task.Lease.Epoch, "expected_version": task.Version})
|
||||
if err := s.Append(domain.Event{ID: "release", Type: "TaskReleased", TaskID: "t", Version: task.Version + 1, Payload: rel, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Lease("t", "h", time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// This is the moment the planning context is rendered.
|
||||
got, _ := s.Task("t")
|
||||
if got.PlanMismatch == nil {
|
||||
t.Fatal("the planner is convened to settle a contradiction it is no longer told about")
|
||||
}
|
||||
if got.PlanMismatch.PhaseID != "phase-2" {
|
||||
t.Fatalf("the contradiction changed across the rotation: %+v", got.PlanMismatch)
|
||||
}
|
||||
}
|
||||
|
||||
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
@@ -0,0 +1,635 @@
|
||||
# orchestra web control ui
|
||||
|
||||
status: mockup specification
|
||||
app: orchestra
|
||||
shared design system: ethos
|
||||
accent: signal violet `#8F7AE5`
|
||||
motif: routing fork
|
||||
primary operator: single infrastructure operator
|
||||
|
||||
## 1. purpose
|
||||
|
||||
orchestra's web ui is an operator console for understanding and steering agent work without reconstructing state from logs, tmux sessions, gitea, or the event stream.
|
||||
|
||||
it has three jobs:
|
||||
|
||||
1. show what needs the operator now.
|
||||
2. show what orchestra believes and what an agent is actually doing.
|
||||
3. expose intervention and forensic detail without making the default screens noisy.
|
||||
|
||||
it is not a generic project-management product, an analytics dashboard, or a chat-first interface.
|
||||
|
||||
## 2. ethos application
|
||||
|
||||
orchestra inherits the ethos shell and neutral system unchanged.
|
||||
|
||||
### fingerprint
|
||||
|
||||
- accent: signal violet `#8F7AE5`
|
||||
- motif: routing fork
|
||||
- accent use: active navigation, focus, selected phase, primary operator action, routing/transition marks
|
||||
- motif use: app mark, phase transitions, routing state, empty states, one hero/detail moment
|
||||
|
||||
suggested token block:
|
||||
|
||||
```css
|
||||
[data-app="orchestra"] {
|
||||
--accent: #8F7AE5;
|
||||
--accent-hi: #A291EC;
|
||||
--accent-dim: rgba(143, 122, 229, 0.14);
|
||||
--accent-line: rgba(143, 122, 229, 0.32);
|
||||
--accent-glow: rgba(143, 122, 229, 0.22);
|
||||
}
|
||||
```
|
||||
|
||||
### type
|
||||
|
||||
- geist sans: labels, task names, explanations, buttons, headings, operator-authored prose
|
||||
- geist mono: task ids, epochs, worker ids, pane ids, timestamps, percentages, context counts, revisions, paths, refs, sha values, lease durations
|
||||
- machine values must never silently use sans for visual convenience
|
||||
|
||||
### surfaces
|
||||
|
||||
- page room: `--bg-0`
|
||||
- normal panel: `--bg-1`
|
||||
- selected/raised content: `--bg-2`
|
||||
- hover: `--bg-3`
|
||||
- borders: 1px `--line`, `--line-hi` only where selection or risk needs stronger separation
|
||||
- no blur or glass
|
||||
- accent never washes a whole content panel
|
||||
|
||||
### motion
|
||||
|
||||
all ui transitions use ethos mechanical motion: 130–170 ms, `cubic-bezier(0.2, 0, 0, 1)`, no spring or bounce.
|
||||
|
||||
## 3. shared shell
|
||||
|
||||
### desktop
|
||||
|
||||
- 64px vertical icon rail
|
||||
- 56px top bar
|
||||
- main content fills remaining viewport
|
||||
- global `cmd+k` command/search surface in the top bar
|
||||
- right side of top bar shows only high-value machine state: orchestra revision, live state, current time
|
||||
|
||||
primary navigation:
|
||||
|
||||
1. dashboard
|
||||
2. tasks
|
||||
3. decisions
|
||||
4. workers
|
||||
5. projects
|
||||
6. review
|
||||
7. settings
|
||||
|
||||
terminal is entered from a task or worker and does not need permanent primary navigation.
|
||||
|
||||
### mobile
|
||||
|
||||
- rail becomes bottom navigation with no more than five primary items
|
||||
- secondary pages move behind task/project drill-down or header menus
|
||||
- machine readouts relocate into detail screens rather than disappearing
|
||||
- terminal remains a dedicated full-screen view
|
||||
|
||||
## 4. information hierarchy
|
||||
|
||||
orchestra should visually distinguish five classes of information.
|
||||
|
||||
| class | meaning | treatment |
|
||||
|---|---|---|
|
||||
| operator authority | human decision, correction, constraint | high prominence, accent signal |
|
||||
| verified orchestra state | lifecycle, lease, phase, review, refs | neutral surfaces, machine values in mono |
|
||||
| agent claims | proposed plan, handoff text, self-reported risk | visibly labeled as agent-supplied |
|
||||
| historical/superseded | stale plan, previous handoff, old decision | muted, never visually dominant |
|
||||
| fault/attention | blocked, refused, stale, mismatch | status color + explicit reason |
|
||||
|
||||
raw events and implementation details are drill-down evidence, not default-page content.
|
||||
|
||||
## 5. dashboard
|
||||
|
||||
mockup: `final/01-dashboard.png`
|
||||
|
||||
purpose: answer "what needs me, what is running, and is the system healthy?"
|
||||
|
||||
### layout
|
||||
|
||||
three primary regions only:
|
||||
|
||||
1. **needs your attention**
|
||||
- trajectory gate
|
||||
- blocking human decision
|
||||
- pr awaiting review
|
||||
- closed-without-merge/operator-required state
|
||||
- one line explaining why each item needs the operator
|
||||
|
||||
2. **running tasks**
|
||||
- task name/id
|
||||
- current phase
|
||||
- worker/harness
|
||||
- last real progress
|
||||
- context occupancy
|
||||
- lease timing
|
||||
- compact routing-fork phase indicator
|
||||
|
||||
3. **system state and capacity**
|
||||
- workers online/available
|
||||
- quota status
|
||||
- active leases
|
||||
- router queue/rejection state
|
||||
- source reconciliation
|
||||
- herdr health
|
||||
|
||||
### rules
|
||||
|
||||
- no disk, cpu, ram, or generic host-monitor metrics unless they directly block orchestration
|
||||
- no vanity charts
|
||||
- health should show actual counts/times, not vague "running" indicators
|
||||
- dashboard actions should navigate to the relevant task/review/worker rather than becoming an operator-control surface itself
|
||||
|
||||
## 6. tasks list
|
||||
|
||||
mockup: `final/04-tasks.png`
|
||||
|
||||
purpose: browse all work under orchestra and locate a task quickly.
|
||||
|
||||
### groups
|
||||
|
||||
- needs attention
|
||||
- running
|
||||
- waiting/queued
|
||||
- completed
|
||||
- failed
|
||||
|
||||
### row fields
|
||||
|
||||
- task title/id
|
||||
- project
|
||||
- current phase
|
||||
- compact routing-fork phase path
|
||||
- worker/harness when assigned
|
||||
- last update/progress
|
||||
- context occupancy for active sessions
|
||||
- status
|
||||
|
||||
### filters
|
||||
|
||||
- project
|
||||
- phase
|
||||
- worker/harness
|
||||
- status
|
||||
- attention state
|
||||
|
||||
avoid turning this into a kanban board. orchestra's lifecycle is stateful and event-backed; the list should expose state, not encourage arbitrary drag-and-drop mutation.
|
||||
|
||||
## 7. task detail
|
||||
|
||||
mockup: `final/02-task-detail.png`
|
||||
|
||||
purpose: provide one calm page answering "what are we doing?", "what is happening?", and "where are we in the workflow?"
|
||||
|
||||
### top area
|
||||
|
||||
- title
|
||||
- task id
|
||||
- project
|
||||
- current phase
|
||||
- steer/correct primary action
|
||||
- secondary actions menu
|
||||
|
||||
### what we're doing
|
||||
|
||||
show only:
|
||||
|
||||
- goal
|
||||
- acceptance summary
|
||||
- latest effective human decisions when present
|
||||
- active constraints when present
|
||||
|
||||
full contract, all decisions, and historical versions open in drill-down views.
|
||||
|
||||
### live execution
|
||||
|
||||
show only:
|
||||
|
||||
- worker/harness
|
||||
- pane
|
||||
- lease remaining
|
||||
- context occupancy
|
||||
- last progress
|
||||
- one-line current activity
|
||||
|
||||
links:
|
||||
|
||||
- open live pane
|
||||
- worker logs
|
||||
- launch context
|
||||
- git diff
|
||||
|
||||
### workflow
|
||||
|
||||
render the ace-fca path visibly:
|
||||
|
||||
`frame → research → plan → implement → review → pr → merged`
|
||||
|
||||
completed phases are quiet green, current phase uses the orchestra accent, future phases remain neutral.
|
||||
|
||||
phase detail, artifacts, refusal history, and raw events belong in drill-downs rather than the default task screen.
|
||||
|
||||
## 8. live pane / terminal
|
||||
|
||||
mockup: `final/03-terminal-live-pane.png`
|
||||
|
||||
purpose: show the actual tmux/herdr pane, not a reconstructed log view.
|
||||
|
||||
### behavior
|
||||
|
||||
- read-only by default
|
||||
- exact pane dimensions and terminal rendering preserved when possible
|
||||
- fit mode may scale visually but must not alter the underlying pane
|
||||
- scrollback inspection must not disturb the live cursor
|
||||
- `take control` explicitly enables keyboard forwarding
|
||||
- strong visual state when control is captured
|
||||
- escape or a fixed shortcut releases control
|
||||
|
||||
### why this screen exists
|
||||
|
||||
it must reveal real tui state such as:
|
||||
|
||||
- input still sitting in the editor
|
||||
- queued input
|
||||
- paste placeholders
|
||||
- permission/approval dialogs
|
||||
- `/clear`
|
||||
- `@HANDOFF.md`
|
||||
- agent busy/idle presentation
|
||||
- tmux status line and pane identity
|
||||
|
||||
### surrounding chrome
|
||||
|
||||
keep it minimal:
|
||||
|
||||
- worker/pane identity
|
||||
- read-only/control state
|
||||
- fit/copy/scrollback controls
|
||||
- lease epoch/timing
|
||||
- context occupancy
|
||||
- last verified progress
|
||||
- pane/worker health
|
||||
- small live event strip for orchestra-originated interactions and confirmations
|
||||
|
||||
when a pane is lost, preserve the last captured frame and state the exact loss time and lease consequence.
|
||||
|
||||
## 9. decisions list
|
||||
|
||||
mockup: `final/05-decisions.png`
|
||||
|
||||
purpose: inspect durable human authority and approvals across tasks.
|
||||
|
||||
this is not the human attention queue; that is review.
|
||||
|
||||
### list fields
|
||||
|
||||
- decision id
|
||||
- short decision text
|
||||
- task
|
||||
- phase
|
||||
- source/provenance
|
||||
- requested/recorded time
|
||||
- status: active, superseded, waiting, approved, rejected, auto
|
||||
- operator/actor
|
||||
|
||||
### filters
|
||||
|
||||
- active/superseded
|
||||
- kind: decision, correction, constraint, answer, scope change
|
||||
- project
|
||||
- task
|
||||
- actor/source
|
||||
|
||||
### detail
|
||||
|
||||
opening a decision should show:
|
||||
|
||||
- exact value
|
||||
- subject
|
||||
- provenance
|
||||
- supersedes/superseded-by links
|
||||
- event sequence
|
||||
- effective authority impact
|
||||
|
||||
machine provenance is mono; the human decision text remains sans.
|
||||
|
||||
## 10. workers
|
||||
|
||||
mockup: `final/06-workers.png`
|
||||
|
||||
purpose: answer "where can work run and is that execution machinery usable?"
|
||||
|
||||
### worker list fields
|
||||
|
||||
- worker/harness identity
|
||||
- online/idle/busy/offline/degraded
|
||||
- harness type
|
||||
- current task when leased
|
||||
- phase
|
||||
- lease or heartbeat age
|
||||
|
||||
intentionally omit generic infrastructure metrics such as disk, memory, cpu, and uptime unless orchestra directly uses them for eligibility.
|
||||
|
||||
### worker detail
|
||||
|
||||
- current session/task
|
||||
- pane id
|
||||
- lease epoch/remaining
|
||||
- last verified progress
|
||||
- worker revision
|
||||
- herdr status
|
||||
- last heartbeat
|
||||
- projects declared by the worker
|
||||
- declared capabilities if orchestra actually consumes them
|
||||
|
||||
### capabilities
|
||||
|
||||
capabilities must come from worker registration/project configuration, never be guessed from host inspection.
|
||||
|
||||
the ui labels them **declared capabilities** and can expose registration provenance.
|
||||
|
||||
### registration
|
||||
|
||||
registration is worker-driven:
|
||||
|
||||
1. worker starts with coordinator credentials/configuration
|
||||
2. worker announces identity, revision, harnesses, projects, capacity and declared capabilities
|
||||
3. coordinator records it and heartbeat updates liveness
|
||||
|
||||
`add worker` should therefore be a registration guide/bootstrap flow, not a magical browser-side creation of a remote process.
|
||||
|
||||
## 11. projects
|
||||
|
||||
mockup: `final/07-projects.png`
|
||||
|
||||
purpose: group long-running repository/workflow configuration and its active work.
|
||||
|
||||
### project list
|
||||
|
||||
- project name
|
||||
- short description
|
||||
- state
|
||||
- active/total task count
|
||||
- attention/decision count
|
||||
- last meaningful activity
|
||||
|
||||
avoid aggregate artifact counts or infrastructure metrics unless they help choose an action.
|
||||
|
||||
### project detail
|
||||
|
||||
- description
|
||||
- source/repository identity
|
||||
- configured worker/harness affinity
|
||||
- workflow phases
|
||||
- trajectory-gate policy
|
||||
- quality gate
|
||||
- human reconcile source
|
||||
- safe-operation policy
|
||||
- active tasks
|
||||
- recent meaningful events
|
||||
|
||||
configuration edits belong in the project detail, while global defaults remain in settings.
|
||||
|
||||
## 12. review
|
||||
|
||||
mockup: `final/08-review.png`
|
||||
|
||||
purpose: the operator inbox: **what needs me to look at something and judge?**
|
||||
|
||||
workers show machinery. review shows human attention.
|
||||
|
||||
### review item types
|
||||
|
||||
- trajectory decision
|
||||
- human blocker / decision request
|
||||
- code/pr review
|
||||
- ai review requiring operator override
|
||||
- operator-required lifecycle state
|
||||
- closed pr without merge
|
||||
- information/fyi, optionally hidden by default
|
||||
|
||||
### list fields
|
||||
|
||||
- type
|
||||
- task/item
|
||||
- why it is here
|
||||
- project
|
||||
- priority
|
||||
- age
|
||||
|
||||
worker/pane is secondary metadata, not a primary column.
|
||||
|
||||
### detail drawer/page
|
||||
|
||||
show the minimum evidence needed to act:
|
||||
|
||||
- why this needs the operator
|
||||
- effective task goal/constraints relevant to the decision
|
||||
- proposed options when applicable
|
||||
- relevant plan/diff/review evidence
|
||||
- provenance and age
|
||||
- comments/context drill-down
|
||||
|
||||
primary action names must match the lifecycle action: make decision, answer blocker, review pr, request changes, retry, etc.
|
||||
|
||||
snooze/defer may exist for attention management but must not silently mutate underlying task truth.
|
||||
|
||||
## 13. settings
|
||||
|
||||
mockup: `final/09-settings.png`
|
||||
|
||||
purpose: configure orchestra itself, project-independent defaults, integrations, policies, and operator/security settings.
|
||||
|
||||
### tabs
|
||||
|
||||
1. general
|
||||
2. appearance
|
||||
3. notifications
|
||||
4. access & security
|
||||
5. integrations
|
||||
6. agents & tools
|
||||
7. policies
|
||||
8. advanced
|
||||
|
||||
### general
|
||||
|
||||
- instance name/environment
|
||||
- locale/time display
|
||||
- default task preferences
|
||||
- default workflow policy
|
||||
- event/artifact retention
|
||||
- storage/backup state only if orchestra owns it
|
||||
|
||||
### appearance
|
||||
|
||||
- theme: system/dark/light
|
||||
- ethos remains fixed; no arbitrary shell redesign controls
|
||||
- accessibility and reduced motion
|
||||
|
||||
### access & security
|
||||
|
||||
- surface tokens/status
|
||||
- trusted review actors
|
||||
- ignored actors
|
||||
- operator sessions
|
||||
- agent-surface auth
|
||||
- secret/config source status without exposing secret values
|
||||
|
||||
### integrations
|
||||
|
||||
- gitea
|
||||
- vikunja
|
||||
- notification sinks
|
||||
- source reconciliation state
|
||||
|
||||
show endpoint/identity and last successful interaction in mono.
|
||||
|
||||
### agents & tools
|
||||
|
||||
- harness definitions
|
||||
- adapter/runtime configuration
|
||||
- tmux/herdr ownership
|
||||
- default launch behavior
|
||||
- context occupancy thresholds
|
||||
|
||||
### policies
|
||||
|
||||
- work phase defaults
|
||||
- trajectory gate
|
||||
- bounded decision request budget
|
||||
- retry policy
|
||||
- reconcile failure handoff threshold
|
||||
- quality/review/submission policy
|
||||
|
||||
### advanced
|
||||
|
||||
- event-store diagnostics
|
||||
- snapshots/replay
|
||||
- raw configuration
|
||||
- migration/status information
|
||||
- destructive operator actions in a visually separated danger zone
|
||||
|
||||
## 14. global command surface
|
||||
|
||||
`cmd+k` is the shared ethos command surface and should handle navigation plus safe operator actions.
|
||||
|
||||
examples:
|
||||
|
||||
- open task by id/title
|
||||
- open worker/pane
|
||||
- show decisions for task
|
||||
- steer/correct active task
|
||||
- answer current blocker
|
||||
- retry failed task
|
||||
- open review item
|
||||
|
||||
high-risk lifecycle actions require explicit confirmation and should not be the first fuzzy-search result for ordinary text.
|
||||
|
||||
## 15. status grammar
|
||||
|
||||
status is compact and consistent across screens.
|
||||
|
||||
recommended semantic colors:
|
||||
|
||||
- green: verified healthy/completed/accepted
|
||||
- violet: current/selected/orchestra-controlled active state
|
||||
- amber: waiting/degraded/operator attention
|
||||
- red: failed/refused/danger
|
||||
- gray: idle/unknown/historical
|
||||
|
||||
color never carries the entire meaning; every state also has text/iconography.
|
||||
|
||||
## 16. evidence drill-down
|
||||
|
||||
any important derived claim should be traceable without ssh.
|
||||
|
||||
from relevant detail screens the operator should be able to reach:
|
||||
|
||||
- event(s) that established the state
|
||||
- exact generated launch context
|
||||
- sealed research/plan/review/submission artifact refs
|
||||
- git sha/ref
|
||||
- worker journal observations
|
||||
- input confirmation receipt
|
||||
- pane capture/terminal
|
||||
- source comment/review provenance
|
||||
|
||||
raw evidence is accessible but not visible by default.
|
||||
|
||||
## 17. copy rules
|
||||
|
||||
- labels describe what the operator controls, not implementation class names
|
||||
- machine values remain exact and honest
|
||||
- errors state what failed and what consequence follows
|
||||
- no "something went wrong"
|
||||
- no spinner where a queue depth, heartbeat age, poll timestamp, retry time, or progress value exists
|
||||
|
||||
examples:
|
||||
|
||||
- `source reconcile failed · retry at 14:06:00`
|
||||
- `worker heartbeat 42s old · ineligible after 60s`
|
||||
- `phase request refused · next phase is research`
|
||||
- `pane lost at 13:42:18 · lease expires in 09:14`
|
||||
|
||||
## 18. responsive behavior
|
||||
|
||||
### tablet/mobile priorities
|
||||
|
||||
keep visible:
|
||||
|
||||
1. attention state
|
||||
2. task title/phase
|
||||
3. latest human authority
|
||||
4. live execution state
|
||||
5. primary action
|
||||
|
||||
collapse or drill down:
|
||||
|
||||
- long acceptance lists
|
||||
- complete workflow history
|
||||
- detailed lease metadata
|
||||
- worker journal
|
||||
- raw events
|
||||
- project configuration
|
||||
|
||||
terminal becomes its own full-screen route and should not be squeezed into a card.
|
||||
|
||||
## 19. implementation order
|
||||
|
||||
recommended web-ui build order:
|
||||
|
||||
1. ethos shell + orchestra tokens/motif
|
||||
2. dashboard
|
||||
3. tasks list
|
||||
4. task detail
|
||||
5. live pane
|
||||
6. review inbox
|
||||
7. decisions list/detail
|
||||
8. workers
|
||||
9. projects
|
||||
10. settings
|
||||
11. global command surface
|
||||
12. evidence/raw-event drill-down
|
||||
|
||||
build each screen against real orchestra api data rather than static ui-only state as early as practical.
|
||||
|
||||
## 20. verification
|
||||
|
||||
for every screen:
|
||||
|
||||
- render and visually inspect pixels
|
||||
- check desktop and <=640px mobile layout
|
||||
- assert no horizontal overflow
|
||||
- verify mono/sans split visibly
|
||||
- verify accent is signal-only
|
||||
- verify no blur/backdrop-filter exists
|
||||
- verify empty/error/loading states show real machine state
|
||||
- compare implemented screen against the corresponding mockup as direction, not pixel-perfect contract
|
||||
|
||||
mockups communicate hierarchy, density, and interaction intent. orchestra state and the ethos design laws are the actual specification.
|
||||