Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 6cb2f932d8 | |||
| b5c37f693b | |||
| bbf3335857 | |||
| 99b209ba10 | |||
| ab3258833d | |||
| c587f2cc8d | |||
| 41658aea5b | |||
| 765bf2afc6 | |||
| cbd6b11c49 |
@@ -2527,3 +2527,324 @@ transaction and no session behind, on either worker.
|
|||||||
|
|
||||||
Both workers hold no sessions and no release transactions. Deployed pair is
|
Both workers hold no sessions and no release transactions. Deployed pair is
|
||||||
still `3c7cf95` on both halves.
|
still `3c7cf95` on both halves.
|
||||||
|
|
||||||
|
## Run 16, 2026-08-29: the rotation rung, half proven
|
||||||
|
|
||||||
|
Task `06G4SWEVP71FYYKAV5FV0ZK5ZG` on `34f3c28`, a three-phase plan
|
||||||
|
(`78d25730140c`) with automated checks inside policy.
|
||||||
|
|
||||||
|
### Proven: a successor inherits the whole sealed plan
|
||||||
|
|
||||||
|
The launch context of the session that picked the task up after its rotation
|
||||||
|
carries the complete plan: overview, current state, desired end state,
|
||||||
|
non-goals, approach, all three phases with their files, changes and
|
||||||
|
verification, testing strategy, risks, migration, and ten research citations.
|
||||||
|
|
||||||
|
### Not a defect: that successor was not given phase progress
|
||||||
|
|
||||||
|
`renderPlanProgress` runs only for `WorkPhaseImplement`
|
||||||
|
(`internal/agentctx/agentctx.go:321`). This successor picked up in review, and
|
||||||
|
the code says why: an independent review must reconstruct the change from the
|
||||||
|
diff rather than inherit the implementer's account. Progress is withheld there
|
||||||
|
deliberately.
|
||||||
|
|
||||||
|
So the rung still needs an **implement-phase** successor. The records exist and
|
||||||
|
are durable, projected by `Task.PlanPhases()`.
|
||||||
|
|
||||||
|
### Why the mid-implement rotation missed
|
||||||
|
|
||||||
|
A trivial task spends about four minutes in implement, and all three phases
|
||||||
|
verified inside two and a half of them, every one against the same tree
|
||||||
|
`3da86dbf8ad5`. The implementer does the whole change first, then verifies each
|
||||||
|
phase. The suspend, edit and restart that sets `handoff_requested` takes about
|
||||||
|
a minute, and it landed after the implementer had already asked for review.
|
||||||
|
|
||||||
|
Two mechanics worth keeping:
|
||||||
|
|
||||||
|
- **The web handoff action cannot drive this.** `RequestHandoff` needs a local
|
||||||
|
coordinator and every session here is worker-owned, so it answers 503. That
|
||||||
|
is the Design A guardrail working, and a fourth UI-exposed gap.
|
||||||
|
- **A state-file edit does not survive a running worker.** It holds sessions in
|
||||||
|
memory and writes them back. Suspend, edit, then restart; resuming lets the
|
||||||
|
old copy win. This cost one attempt to learn.
|
||||||
|
|
||||||
|
### F62: a requested handoff that is never answered is invisible
|
||||||
|
|
||||||
|
The rotation is agent-driven: the worker asks, and the agent must write
|
||||||
|
`HANDOFF.md` at a turn boundary. The review agent never did. Renewals stopped,
|
||||||
|
the lease expired at 10:12:05, and the task requeued having lost an attempt.
|
||||||
|
|
||||||
|
Worker health recorded exactly one thing, 34 times:
|
||||||
|
|
||||||
|
```text
|
||||||
|
x34 lease 06G4SWEVP71FYYKAV5FV0ZK5ZG not renewed: agent status idle and pane unchanged
|
||||||
|
```
|
||||||
|
|
||||||
|
Nothing says a handoff was requested and left unanswered. There is no timeout,
|
||||||
|
no retry, and no observation. The only visible consequence is an expiry that
|
||||||
|
looks identical to an ordinary idle one. Diagnosing it needed the worker state
|
||||||
|
file.
|
||||||
|
|
||||||
|
The ring earned its keep again: one distinct message with a count of 34, rather
|
||||||
|
than 34 overwrites of one slot.
|
||||||
|
|
||||||
|
**Fixed, not yet proven live.** The request is now stamped
|
||||||
|
(`herdr.Session.HandoffRequestedAt`) and the wait is bounded by
|
||||||
|
`watchHandoff` in the worker:
|
||||||
|
|
||||||
|
- the lease renews while Orchestra is explicitly waiting, because a quiet pane
|
||||||
|
is the answer the agent was asked for — the renewal gate's new case, bounded
|
||||||
|
by `handoffAnswerTimeout`;
|
||||||
|
- the request is re-sent once at `handoffRetryAfter` (4 minutes), with the same
|
||||||
|
reason it was first asked with;
|
||||||
|
- at 10 minutes the worker nacks with failure class `handoff_unanswered`, and
|
||||||
|
the coordinator emits `TaskReleased reason=handoff_unanswered` rather than
|
||||||
|
letting the lease die as generic idleness.
|
||||||
|
|
||||||
|
`DebtClassForFailureClass` knows the class, so a harness that repeatedly
|
||||||
|
ignores handoff requests now accumulates in the debt ledger instead of hiding
|
||||||
|
inside `lease_expired`.
|
||||||
|
|
||||||
|
Runtime proof still owed: force a request the agent will not answer, and read
|
||||||
|
the release event rather than the worker state file.
|
||||||
|
|
||||||
|
## Run 17, 2026-08-29: F62 proven live on `c587f2c`
|
||||||
|
|
||||||
|
Task `06G4V20T528ZTER7KZBVGNAYXC`, a five-phase plan
|
||||||
|
(`3b16b77054b9`) on the deployed pair, coordinator and worker both at
|
||||||
|
`c587f2c`.
|
||||||
|
|
||||||
|
### The trigger was production, not a rig
|
||||||
|
|
||||||
|
The rig planned for this run was a state-file lever. It was never needed. The
|
||||||
|
`implement -> review` phase change asked for a handoff through the ordinary
|
||||||
|
path (`rotateForPhase`, reason `phase_changed`), and the agent ignored it
|
||||||
|
because this task's brief instructed it to from phase 4 onward. Nothing touched
|
||||||
|
tmux, no process was stopped, and the pane stayed open for the whole wait.
|
||||||
|
|
||||||
|
```text
|
||||||
|
12:48:09 implement -> review, handoff requested, reason=phase_changed
|
||||||
|
12:50:09 TaskLeaseRenewed agent idle, pane capture unchanged
|
||||||
|
12:52:15 pane input confirmed the single resend, 4m06s after the request
|
||||||
|
12:52:39 TaskLeaseRenewed
|
||||||
|
12:55:14 TaskLeaseRenewed
|
||||||
|
12:57:49 TaskLeaseRenewed
|
||||||
|
12:58:14 TaskReleased reason=handoff_unanswered failure_class=handoff_unanswered
|
||||||
|
last_error="handoff requested (phase_changed) and unanswered for 10m5s"
|
||||||
|
12:59:14 TaskLeased successor leases normally
|
||||||
|
13:02:20 TaskSubmitted
|
||||||
|
```
|
||||||
|
|
||||||
|
Every assertion the operator set for this rung held:
|
||||||
|
|
||||||
|
- the release does not look like an ordinary `lease_expired`: it carries
|
||||||
|
`reason=handoff_unanswered` and `lifecycle_phase=handoff_unanswered`;
|
||||||
|
- the idle-renewal rule did not kill the wait: four renewals on an idle pane
|
||||||
|
with no capture movement, each one the gate would have refused before this
|
||||||
|
commit;
|
||||||
|
- the resend preserved the reason the request was first made with;
|
||||||
|
- exactly one resend, not one per tick;
|
||||||
|
- the timeout produced the class at `10m5s`, one tick past the bound;
|
||||||
|
- nothing stale was left: `releases: {}`, no session, no lease, and zero tmux
|
||||||
|
panes matching the id;
|
||||||
|
- the next attempt leased 60s later and reached `TaskSubmitted`.
|
||||||
|
|
||||||
|
The debt projection classified it with no new code beyond the class mapping:
|
||||||
|
|
||||||
|
```text
|
||||||
|
v1:operational:handoff_unanswered:workpc-claude:lease operational important
|
||||||
|
"handoff requested (phase_changed) and unanswered for 10m5s"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Inference, not observation.** The resent prompt's text was not captured. That
|
||||||
|
it carried the `phase_changed` wording rests on the code path and the timing,
|
||||||
|
not on a pane capture. A future rig should publish a capture across the resend.
|
||||||
|
|
||||||
|
### The implement-successor rung missed again, and why
|
||||||
|
|
||||||
|
A five-phase plan does not lengthen the implement phase. This one ran 12:45:18
|
||||||
|
to 12:48:09, about 2m50s, with verifications 20 to 40 seconds apart:
|
||||||
|
|
||||||
|
```text
|
||||||
|
12:45:47 phase-1 12:46:07 phase-2 12:46:32 phase-3
|
||||||
|
12:47:12 phase-4 12:47:52 phase-5 12:48:09 implement -> review
|
||||||
|
```
|
||||||
|
|
||||||
|
The brief's explicit one-phase-at-a-time requirement was followed, and it
|
||||||
|
still took under three minutes. Phase count is not the lever; volume of
|
||||||
|
mechanical work is.
|
||||||
|
|
||||||
|
`awaiting_manual_verification` does not hold the phase open either. All five
|
||||||
|
phases carried that status and the agent moved on regardless.
|
||||||
|
|
||||||
|
### The state-file lever cannot be driven with `restart` alone
|
||||||
|
|
||||||
|
The documented procedure is stop, edit, start. The unattended sudo rule grants
|
||||||
|
`install` and `systemctl restart orchestra-worker`, with no `stop` or `start`.
|
||||||
|
With `restart` only, the running worker's five-second tick save landed between
|
||||||
|
the edit and the restart, and the edit was gone. Widening the sudoers line is
|
||||||
|
the fix.
|
||||||
|
|
||||||
|
### One cosmetic note
|
||||||
|
|
||||||
|
`Session.HandoffRequestedAt` serializes as `0001-01-01T00:00:00Z` when unset,
|
||||||
|
because `omitempty` does not omit a zero `time.Time`. The worker parses it back
|
||||||
|
to a zero value and `watchHandoff` starts the clock, so behaviour is correct.
|
||||||
|
It is the same marshalling trap the operator console hit.
|
||||||
|
|
||||||
|
## Run 18, 2026-08-29: the implement-successor rung, proven
|
||||||
|
|
||||||
|
Task `06G4VF5HZW7Q4JBM3TTY7W1Y64` on `c587f2c`, a five-phase plan whose phases
|
||||||
|
are ten named checks rather than one refactor. Volume of mechanical work is
|
||||||
|
what keeps the implement phase open; phase count does not.
|
||||||
|
|
||||||
|
### The rotation
|
||||||
|
|
||||||
|
```text
|
||||||
|
13:43:10 implement session launched
|
||||||
|
13:43:44 phase-1 verified
|
||||||
|
13:44:04 phase-2 verified
|
||||||
|
13:44:35 handoff requested, reason=milestone, delivered by watchHandoff's resend
|
||||||
|
13:45:25 TaskLeaseRenewed
|
||||||
|
13:46:51 TaskReleased handoff_ref 0b4fae6f705a anchor 8ea3c7d1ae8f
|
||||||
|
13:46:51 TaskLeased successor, same harness
|
||||||
|
13:46:55 TaskPickupValidated
|
||||||
|
13:47:06 TaskLaunchAcknowledged, still in implement
|
||||||
|
```
|
||||||
|
|
||||||
|
The lever was the worker state file, backdated past `handoffRetryAfter` so the
|
||||||
|
request went out on the next tick. Under `c587f2c` that lever now produces a
|
||||||
|
real prompt: the flag alone used to sit there unasked, which is how run 16
|
||||||
|
ended in a silent expiry.
|
||||||
|
|
||||||
|
### What the successor was given
|
||||||
|
|
||||||
|
From `.orchestra/launch.md`, written at 13:47:05:
|
||||||
|
|
||||||
|
```text
|
||||||
|
## Current phase
|
||||||
|
implement: Implement the accepted plan below. Verify as you go.
|
||||||
|
|
||||||
|
## Verified git state
|
||||||
|
- head: 8ea3c7d1ae8fd960897344694645117adbd8ee82
|
||||||
|
- uncommitted changes: false
|
||||||
|
|
||||||
|
## Current human decisions
|
||||||
|
None recorded. Work from the goal and acceptance above.
|
||||||
|
|
||||||
|
## Plan progress
|
||||||
|
Orchestra established this by running the plan's own verification. You cannot
|
||||||
|
write it.
|
||||||
|
- phase-1 (...): automated checks passed at d3acd4989a20, stale because the
|
||||||
|
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
|
||||||
|
- phase-2 (...): automated checks passed at d3acd4989a20, stale because the
|
||||||
|
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
|
||||||
|
- phase-3 (Checks 7 to 8): not started
|
||||||
|
- phase-4 (...): not started
|
||||||
|
- phase-5 (...): not started
|
||||||
|
```
|
||||||
|
|
||||||
|
Everything the rung asked for is there: the complete accepted plan with all
|
||||||
|
five phases, their files, changes and verification; the phases already
|
||||||
|
verified; the first unfinished phase; and the current human authority. The
|
||||||
|
launch context also carries the accepted research and its dead ends.
|
||||||
|
|
||||||
|
**SHA staleness renders itself.** Neither phase is reported as simply passed.
|
||||||
|
Each says the checks passed at `d3acd4989a20` and are stale because the tree
|
||||||
|
has moved to `8ea3c7d1ae8f`. That is half of the manual-verification rung
|
||||||
|
observed without being asked for.
|
||||||
|
|
||||||
|
### The lever still loses a race, sometimes
|
||||||
|
|
||||||
|
`systemctl restart` alone leaves a five-second window in which the running
|
||||||
|
worker's tick save can clobber the edit. It clobbered run 17's attempt and
|
||||||
|
survived run 18's. Stop, edit, start is the reliable sequence, and it needs
|
||||||
|
`systemctl stop` and `start` in the unattended sudo rule.
|
||||||
|
|
||||||
|
## Run 19, 2026-08-29: the manual verification seam, and F63
|
||||||
|
|
||||||
|
Two tasks, both ingested from Gitea so that comments reconcile:
|
||||||
|
`06G4W2TMZ9MW9SEBM5MGTM5288` (issue 24, plan `500464b9`) and
|
||||||
|
`06G4W63545T3RV8SSKGKN11B3G` (issue 26, plan `53f7918b`).
|
||||||
|
|
||||||
|
A burn-in task cannot test this rung. Comments reconcile only for the source
|
||||||
|
the task came from, so a task created with `source: "burnin"` has no comment
|
||||||
|
stream at all.
|
||||||
|
|
||||||
|
### Proven: the keyed seam
|
||||||
|
|
||||||
|
```text
|
||||||
|
15:09:15 PlanPhaseVerified phase-1 awaiting_manual_verification at 80728967
|
||||||
|
comment: "Looks good to me. Nice work on this one, ship it."
|
||||||
|
17:10:03 HumanDecisionRecorded subject=operator_instruction
|
||||||
|
phase-1 unchanged, still awaiting_manual_verification
|
||||||
|
comment: "orchestra verify phase-1"
|
||||||
|
17:10:25 HumanDecisionRecorded subject=plan_phase_verification:500464b9…:phase-1
|
||||||
|
phase-1 verified
|
||||||
|
```
|
||||||
|
|
||||||
|
All five phases reached `awaiting_manual_verification` on their automated
|
||||||
|
checks alone. A generic approval satisfied nothing. The keyed form flipped
|
||||||
|
exactly the phase it named, under the plan ref Orchestra supplied rather than
|
||||||
|
one the commenter chose.
|
||||||
|
|
||||||
|
### Proven: re-verification at a new SHA
|
||||||
|
|
||||||
|
On issue 26, inside one live implement lease, with the worker stopped for the
|
||||||
|
whole edit:
|
||||||
|
|
||||||
|
```text
|
||||||
|
15:23:00 PlanPhaseVerified phase-1 awaiting_manual_verification at f5d8f90b
|
||||||
|
operator commit moves the tree f5d8f90b -> 1f07b89c
|
||||||
|
.orchestra/plan-progress.json requests phase-1 again
|
||||||
|
15:23:17 PlanPhaseVerified phase-1 awaiting_manual_verification at 1f07b89c
|
||||||
|
17:23:16 HumanDecisionRecorded subject=plan_phase_verification:53f7918b…:phase-1
|
||||||
|
phase-1 verified at 1f07b89c
|
||||||
|
```
|
||||||
|
|
||||||
|
The rerun ran the plan's own commands at the new tree and wrote a fresh record
|
||||||
|
bound to it. The worker never reads a command out of the request: the request
|
||||||
|
names a phase and nothing else about it is trusted.
|
||||||
|
|
||||||
|
### F63: a manual sign-off outlives the tree it was given against
|
||||||
|
|
||||||
|
The second half of the same rig, with no new comment:
|
||||||
|
|
||||||
|
```text
|
||||||
|
operator commit moves the tree 1f07b89c -> 28af84ff
|
||||||
|
.orchestra/plan-progress.json requests phase-1 again
|
||||||
|
15:24:02 PlanPhaseVerified phase-1 verified at 28af84ff
|
||||||
|
```
|
||||||
|
|
||||||
|
`RecordPlanPhaseVerification` consults `manuallySignedOff`
|
||||||
|
(`internal/operations/planprogress.go:127,152`), which asks only whether a
|
||||||
|
decision with that subject exists. It does, forever, for that plan ref and
|
||||||
|
phase id. So the automated half of a phase is re-established at every new SHA
|
||||||
|
while the manual half is asserted from a tree the human never saw.
|
||||||
|
|
||||||
|
This is the same class of bug `AtSHA` exists to prevent, applied to the half
|
||||||
|
that `AtSHA` does not cover. A manual check on this project is a human reading
|
||||||
|
printed output; an edit between the sign-off and the rerun can change exactly
|
||||||
|
that output.
|
||||||
|
|
||||||
|
Only a replan clears it, because `PlanPhases()` drops records whose plan ref is
|
||||||
|
not the accepted one.
|
||||||
|
|
||||||
|
Proposed shape, not built: when the reducer flips a record on a keyed decision
|
||||||
|
(`internal/store/store.go:273`), it knows the record it satisfied and that
|
||||||
|
record's `AtSHA`. Stamp that sha on the decision, and have `manuallySignedOff`
|
||||||
|
require it to match the run being recorded. A sign-off that arrives before any
|
||||||
|
run has no sha and keeps today's behaviour, so the ordinary ordering is
|
||||||
|
unchanged.
|
||||||
|
|
||||||
|
### Also established
|
||||||
|
|
||||||
|
- `review -> implement` is legal in `legalPhaseTransitions`, but
|
||||||
|
`AdvanceWorkPhase` only walks the project's path forward. A task that has
|
||||||
|
left implement can return only through a review that requests changes, not
|
||||||
|
through the operator phase endpoint.
|
||||||
|
- The agent's re-verification request is a file an operator can write:
|
||||||
|
`.orchestra/plan-progress.json`, `{"phase": "...", "status":
|
||||||
|
"ready_for_verification"}`. It is the whole lever for this rung.
|
||||||
|
- Ingest turns every issue label into a required capability, so a rig issue
|
||||||
|
must carry no labels or it will not schedule.
|
||||||
|
|||||||
@@ -0,0 +1,349 @@
|
|||||||
|
# Handoff: the release path is settled, and the UI became a truth detector
|
||||||
|
|
||||||
|
Written 2026-08-29, 14:20 local (10:20 UTC). Read with `BURNIN.md` (the run
|
||||||
|
ledger, current through run 16), `DEBT-DESIGN.md`, `PLAN-SPEC-DESIGN.md`,
|
||||||
|
`AUDIT.md` and `CLAUDE.md`. The previous handoff is
|
||||||
|
`HANDOFF-2026-08-28-plan-v1.md`.
|
||||||
|
|
||||||
|
Everything below was observed live unless it says otherwise.
|
||||||
|
|
||||||
|
## The headline
|
||||||
|
|
||||||
|
**The expired-release defect is fixed and proven both ways.** The whole family
|
||||||
|
around it is closed. F57 through F60 settle what happens to a release
|
||||||
|
transaction in every case. That includes the ones that used to need an operator
|
||||||
|
with a text editor.
|
||||||
|
|
||||||
|
**Three new things exist that did not before.** A bounded observation ring on
|
||||||
|
worker health, closing F18. A read-only debt ledger projected from the event
|
||||||
|
log. An operator console rebuilt on the ethos design system.
|
||||||
|
|
||||||
|
**The UI turned out to be a truth detector.** Nine screens were built against
|
||||||
|
real endpoints. They found four places where Orchestra has no capability to
|
||||||
|
support the intended interface. That list is the most valuable output of the
|
||||||
|
session.
|
||||||
|
|
||||||
|
```text
|
||||||
|
orchestra-plan-v1 plan machinery proven
|
||||||
|
↓
|
||||||
|
orchestra-f18-baseline d6ee10f, the bounded observation ring
|
||||||
|
↓ 6 commits
|
||||||
|
34f3c28 deployed now: release path settled, debt ledger, new UI
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployed state
|
||||||
|
|
||||||
|
| Half | Revision |
|
||||||
|
|---|---|
|
||||||
|
| Coordinator, homesrv container | `34f3c28` |
|
||||||
|
| Worker, workpc systemd | `34f3c28` |
|
||||||
|
|
||||||
|
```text
|
||||||
|
commit 34f3c2888fc7d45d190d93e1ea42501b6cd3e474
|
||||||
|
coordinator sha256 1d32d83ec859b36e12473fab01bbfc3c769b97b4f6ae4db2841db5161e0eda19
|
||||||
|
worker sha256 0e3877321dea8a1eeda51ccc6f3ead1a14aa5f5cae4d95b704f61c364248ec65
|
||||||
|
```
|
||||||
|
|
||||||
|
`cbd6b11` is one commit above and is documentation only. Three commits are
|
||||||
|
unpushed.
|
||||||
|
|
||||||
|
**Worker installs no longer need a human.** The operator installed the
|
||||||
|
`/etc/sudoers.d` line, so `sudo -n install …` and
|
||||||
|
`sudo -n systemctl restart orchestra-worker` both work unattended. Verify the
|
||||||
|
running revision from the journal, never the installed file.
|
||||||
|
|
||||||
|
## The defects fixed, and how each was found
|
||||||
|
|
||||||
|
Not one came from reading code. Every one came from a live run failing.
|
||||||
|
|
||||||
|
| Id | Commit | What |
|
||||||
|
|---|---|---|
|
||||||
|
| F57 | `6565b9f` | An expired lease could never commit the anchor it had already pushed. The worker sent an epoch the expiry replay had deleted, and the coordinator refused any handoff without a live lease. The epoch now belongs to the transaction, `TaskReleased` retains the ending epoch, and `lateHandoffAccepted` lets exactly that owner commit while the task is queued and unleased. |
|
||||||
|
| F58 | `03663f4` | A superseded transaction retried a permanent 409 forever, holding the pane and pinning `ActiveTask`. Run 10's task did it for seven hours. `TaskLeased` now abandons a transaction whose id the lease does not carry. |
|
||||||
|
| F59 | `8e37989` | F58 fires on `TaskLeased`, and a failed task is never leased again. `TaskFailed` now drops the transaction too. |
|
||||||
|
| F60 | `3c7cf95` | The general rule the other two were reaching for. Terminal is failed or completed. Blocked keeps the transaction, because a reopen can still commit it, so `TaskBlocked` now retains the ending epoch as well. A refusal parks the commit for 30s backing off to 5 minutes, and any event about the task un-parks it. A transport failure is not an answer and retries at once. |
|
||||||
|
| F18 | `d6ee10f` | The single `last_error` slot. Worker health now carries up to sixteen distinct observations with repeat counts and first/last times, collapsing by message rather than by position. |
|
||||||
|
|
||||||
|
### The rig that proved F57, and the guard
|
||||||
|
|
||||||
|
```text
|
||||||
|
19:00:10.742 transaction opens at prepared, anchor pushing
|
||||||
|
19:00:10.727 TaskReleased v15 reason=lease_expired surface=tui
|
||||||
|
19:00:11.662 TaskReleased v16 the late commit, accepted 935ms after the lease died
|
||||||
|
19:00:11.665 TaskLeased v17 successor picks up the handoff
|
||||||
|
19:00:15.524 TaskPickupValidated v18
|
||||||
|
```
|
||||||
|
|
||||||
|
Race guard, next boundary: force the expiry, then lease the task to a probe
|
||||||
|
harness before the push finishes. The late commit is refused, no handoff is
|
||||||
|
written, and the successor's lease stands.
|
||||||
|
|
||||||
|
**The rig technique matters more than the rig.** Suspending the worker cannot
|
||||||
|
produce this ordering. The event replay runs at the top of every tick and
|
||||||
|
discards the transaction. The ordering exists only inside one call:
|
||||||
|
transaction opened, anchor pushing, commit not yet sent. So poll the worker
|
||||||
|
state file at 2ms and fire `POST /v1/tasks/<id>/release` the instant a
|
||||||
|
transaction appears at `prepared`.
|
||||||
|
|
||||||
|
**A named probe harness owns a lease without starting an agent.**
|
||||||
|
`race-guard-probe` never picks anything up and expires on the normal TTL.
|
||||||
|
|
||||||
|
## Corrections to the previous handoff
|
||||||
|
|
||||||
|
**The operator lifecycle actions do not lose a version race.** On a leased
|
||||||
|
task, `block`, `release` and `attention` are refused by
|
||||||
|
`internal/store/store.go:885-901` when the payload omits `harness_id` and
|
||||||
|
`lease_epoch`. Ten attempts in 550ms all failed that way. Send both fencing
|
||||||
|
fields and they succeed on the first try.
|
||||||
|
|
||||||
|
**The OpenCode Zen free tier is not blocked.** The selected model was.
|
||||||
|
|
||||||
|
## The debt ledger
|
||||||
|
|
||||||
|
`DEBT-DESIGN.md` answers nine design questions and carries four amendments the
|
||||||
|
operator made. Slice one is built, deployed and run.
|
||||||
|
|
||||||
|
**Slice one writes nothing.** A read-only projection over the existing log,
|
||||||
|
plus a pure eligibility function and `GET /v1/debt`. It folded 881 events and
|
||||||
|
produced 15 candidates and 3 gaps.
|
||||||
|
|
||||||
|
```text
|
||||||
|
v1:operational:lease_expired:workpc-opencode:lease r=41 tasks=4
|
||||||
|
v1:operational:lease_expired:workpc-claude:lease r=29 tasks=13
|
||||||
|
v1:operational:lease_failure:-:lease r=20 tasks=16
|
||||||
|
v1:correctness:handoff_validation:-:lease r=5 tasks=5
|
||||||
|
```
|
||||||
|
|
||||||
|
The opencode failure shape is the top item, found mechanically. Run 14 reached
|
||||||
|
the same conclusion by hand from a pane capture.
|
||||||
|
|
||||||
|
**It reported what it cannot see, which was the point.** The 409 release loop
|
||||||
|
does not appear. That evidence lived in the F18 worker ring, and no event
|
||||||
|
carries it. Manual interventions are a non-durable gap for the same reason.
|
||||||
|
|
||||||
|
**The first run exposed four defects in the model**, all recorded at the end of
|
||||||
|
`DEBT-DESIGN.md`:
|
||||||
|
|
||||||
|
- the component part is too coarse for lease evidence
|
||||||
|
- `harness` is often empty on block reasons
|
||||||
|
- path normalization mangled a mismatch reference
|
||||||
|
- recurrence alone is the wrong sort order
|
||||||
|
|
||||||
|
Layout, so the read model does not end up in the command layer:
|
||||||
|
|
||||||
|
```text
|
||||||
|
internal/domain/debt.go types, signature, classification
|
||||||
|
internal/store/debt_projection.go the fold, and the gap report
|
||||||
|
internal/operations/debt.go CheckDebtEligibility
|
||||||
|
```
|
||||||
|
|
||||||
|
## The operator console
|
||||||
|
|
||||||
|
Nine screens on the ethos system, signal violet `#8F7AE5`, routing fork motif.
|
||||||
|
Each screen was built by its own agent against a foundation with one author.
|
||||||
|
The shell, tokens and primitives could not drift into nine dialects.
|
||||||
|
|
||||||
|
**Render before signing off.** Three bugs existed that no computed value would
|
||||||
|
have caught. All three came from looking at a screenshot:
|
||||||
|
|
||||||
|
- The previous stylesheet fought every shared class name and leaked properties
|
||||||
|
the new rules never mention, which is how `position: fixed` survived on
|
||||||
|
`.topbar`. It is now scoped under `.legacy` and reaches only the login route.
|
||||||
|
That also stops its green accent and its `backdrop-filter` from reaching the
|
||||||
|
console.
|
||||||
|
- Go marshals a zero `time.Time` as `0001-01-01T00:00:00Z` and `omitempty` does
|
||||||
|
not omit a struct, so absent timestamps arrived populated-looking and
|
||||||
|
rendered as `739855d ago`. Stripped once in `client.ts`, with a test.
|
||||||
|
- Long machine ids overflowed their cards and painted under the next one.
|
||||||
|
|
||||||
|
Chromium is installed at `/usr/bin/chromium`. To see a screen without a live
|
||||||
|
session, write a throwaway harness that stubs `window.fetch` and renders
|
||||||
|
`<Console>` inside a `MemoryRouter`, served by vite on a spare port. Note that
|
||||||
|
`npx` and `./node_modules/.bin/*` do not work on this filesystem: call
|
||||||
|
`node ./node_modules/vite/bin/vite.js` directly.
|
||||||
|
|
||||||
|
## What the UI proved Orchestra cannot do
|
||||||
|
|
||||||
|
This is the part worth acting on. Each screen refused to fake something, and
|
||||||
|
the refusals name real capability gaps.
|
||||||
|
|
||||||
|
| Gap | Evidence |
|
||||||
|
|---|---|
|
||||||
|
| **No web-facing human-decision write path** | `Steer / Correct` is disabled. `internal/ui/ui.go`'s action switch has grant/deny approval, resubmit, handoff, release, block and complete, and nothing writes `HumanDecisionRecorded`. The spec makes steering the primary action of the task detail screen. |
|
||||||
|
| **No keystroke forwarding** | `Take control` is disabled. Only resubmit and approval grant/deny reach a live pane. |
|
||||||
|
| **Context occupancy is trapped in herdr** | Three screens independently hit it. No projection carries it. |
|
||||||
|
| **Project configuration is not served** | Repo, remote, quality gate and verification policy live only in `config.jsonc`. The projects screen can show none of it. |
|
||||||
|
| **The web cannot request a handoff for a federated task** | `RequestHandoff` needs a local coordinator and answers 503. That is the Design A guardrail working. |
|
||||||
|
|
||||||
|
The operator's direction on these. Treat first-class direct human input as the
|
||||||
|
highest-value backend feature. Build it as `POST /v1/tasks/<id>/decisions`,
|
||||||
|
using the same durable decision semantics as Gitea comments, so Gitea, CLI and
|
||||||
|
web converge on one `HumanDecisionRecorded`. Keep take-control disabled,
|
||||||
|
because arbitrary pane input bypasses the durable authority model. Expose
|
||||||
|
occupancy through a session health projection rather than teaching the web
|
||||||
|
server about herdr. Add a read-only effective project configuration endpoint,
|
||||||
|
which F61 will also need.
|
||||||
|
|
||||||
|
## Where the plan-machinery ladder stands
|
||||||
|
|
||||||
|
Proven in run 14: the worker executes the **sealed plan's** commands rather
|
||||||
|
than the request's, and every `PlanPhaseVerified` binds `plan_ref`, `phase_id`,
|
||||||
|
`at_sha`, `evidence_ref`, `lease_epoch` and `harness_id`.
|
||||||
|
|
||||||
|
Proven in run 16: a successor inherits the **whole sealed plan**, all phases
|
||||||
|
with their files, changes, verification, and the research citations.
|
||||||
|
|
||||||
|
**Still unproven, and the next runtime item:**
|
||||||
|
|
||||||
|
```text
|
||||||
|
mid-implement rotation
|
||||||
|
→ successor picks up in implement
|
||||||
|
→ launch context states which phases are already verified
|
||||||
|
|
||||||
|
manual verification
|
||||||
|
→ SHA goes stale
|
||||||
|
→ re-verification
|
||||||
|
|
||||||
|
plan mismatch
|
||||||
|
→ human decision
|
||||||
|
→ real replan, old plan retained, replacement launched
|
||||||
|
```
|
||||||
|
|
||||||
|
Two things make the first one hard, and both are now known:
|
||||||
|
|
||||||
|
- **A trivial task spends about four minutes in implement**, and verifies every
|
||||||
|
phase against one tree near the end. The implementer writes the whole change
|
||||||
|
first, then verifies each phase in turn. Use a task whose implement phase
|
||||||
|
genuinely runs long.
|
||||||
|
- **A state-file edit does not survive a running worker.** It holds sessions in
|
||||||
|
memory and writes them back. Suspend, edit, then `sudo systemctl restart`.
|
||||||
|
Resuming lets the old copy win. Setting `handoff_requested` on the session is
|
||||||
|
the production rotation lever.
|
||||||
|
|
||||||
|
Progress is withheld from a **review** successor on purpose
|
||||||
|
(`internal/agentctx/agentctx.go:321`), because an independent review must
|
||||||
|
reconstruct the change from the diff. That absence is not the defect.
|
||||||
|
|
||||||
|
## F61 and F62, recorded and not built
|
||||||
|
|
||||||
|
**F61: the planner learns the verification policy by refusal.** The brief says
|
||||||
|
a policy exists, not what is in it. Every plan therefore pays one refused round
|
||||||
|
trip. The recovery loop works: run 14's planner consumed the refusal and
|
||||||
|
resealed 31 seconds later. It is not a one-liner, because `agentctx.Input.Policy` is filled
|
||||||
|
from the worker's `SafeOperations` while the verification policy is
|
||||||
|
coordinator-side. The reason to promote it later is local models, which may
|
||||||
|
propose forbidden commands repeatedly because they cannot infer the allowed
|
||||||
|
substitute.
|
||||||
|
|
||||||
|
**F62: a requested handoff nobody answers is invisible.** The rotation is
|
||||||
|
agent-driven. In run 16 the agent never wrote `HANDOFF.md`, renewals stopped,
|
||||||
|
the lease expired, and the task lost an attempt. Worker health recorded only
|
||||||
|
`agent status idle and pane unchanged`, 34 times. There is no timeout, no
|
||||||
|
retry, and no observation saying a handoff was requested and left unanswered.
|
||||||
|
The expiry is indistinguishable from an ordinary idle one.
|
||||||
|
|
||||||
|
## Live state
|
||||||
|
|
||||||
|
Three pull requests are open and unreviewed:
|
||||||
|
|
||||||
|
```text
|
||||||
|
06G4M6HF1Z3EREX1X3NEKSHP24 pulls/18
|
||||||
|
06G4M8WHGQ4P3GQMPEEH0RJRHM pulls/19
|
||||||
|
06G4SWEVP71FYYKAV5FV0ZK5ZG pulls/20
|
||||||
|
```
|
||||||
|
|
||||||
|
Both workers are online on `34f3c28` with no release transactions and no
|
||||||
|
sessions. 29 blocked `test-e2e` tasks are burn-in debris. Three `correx` tasks
|
||||||
|
are queued and unschedulable, because `correx` has no entry in the
|
||||||
|
coordinator's `config.jsonc`.
|
||||||
|
|
||||||
|
**opencode now runs.** The model was the problem, and `hy3-free` works. It then
|
||||||
|
stops on a permission prompt, because `~/.config/opencode/opencode.jsonc` sets
|
||||||
|
`"bash": "ask"`. Orchestra can answer that exact dialog, but only when an
|
||||||
|
operator queues `grant_approval`, so unattended work stalls on the first
|
||||||
|
command. Set `"bash": "allow"` for unattended runs. That file also has no
|
||||||
|
top-level `model` key, so OpenCode picks whatever sits at the top of
|
||||||
|
`~/.local/state/opencode/model.json`, which any manual pick silently changes.
|
||||||
|
|
||||||
|
**The opencode adapter is now a debt item, not a curiosity.** It cannot resolve
|
||||||
|
a session file. Activity therefore reads `unknown`, and the worker cannot tell
|
||||||
|
finished from never-started. The debt projection surfaced it independently as
|
||||||
|
the top recurring operational item.
|
||||||
|
|
||||||
|
## Things that will bite
|
||||||
|
|
||||||
|
- **Background python tasks get killed here.** Three watchers died before doing
|
||||||
|
anything. Foreground polling and the `Monitor` tool both work.
|
||||||
|
- **`/v1/events` is one line of JSON.** A `grep` for two substrings matches
|
||||||
|
across unrelated tasks. Parse it.
|
||||||
|
- **`npx` and `./node_modules/.bin/*` fail on this filesystem.** Call node
|
||||||
|
directly.
|
||||||
|
- **`rm` and `cp` are interactive.** Use `/bin/rm -f` and `install`.
|
||||||
|
- **Secrets are guarded.** Expand a token inside the container in one remote
|
||||||
|
command:
|
||||||
|
`T=$(docker exec orchestra-api printenv ORCHESTRA_TUI_TOKEN); curl -s -H "X-Orchestra-Surface: tui" -H "Authorization: Bearer $T" ...`
|
||||||
|
- **Rebuild the coordinator from a detached worktree**, and split the worktree
|
||||||
|
add, the docker build and the compose up into separate commands.
|
||||||
|
- **`deploy/build.sh` builds both halves from one commit with one stamp.** Use
|
||||||
|
it.
|
||||||
|
- **Geist is not on disk.** Both stacks fall back to system faces, and the
|
||||||
|
ethos threat model rules out the font CDN.
|
||||||
|
|
||||||
|
## Resume in this order
|
||||||
|
|
||||||
|
Set by the operator at the session boundary. Do not rebuild it from the commits.
|
||||||
|
|
||||||
|
1. **F62 first.** Make a requested-but-unanswered handoff visible and bounded.
|
||||||
|
Preserve the lease while Orchestra is explicitly waiting, retry the confirmed
|
||||||
|
request, then emit a causal timeout. Today it ends as generic idleness.
|
||||||
|
2. **Repeat the implement-successor rung** with a deliberately longer task. Get
|
||||||
|
at least one `PlanPhaseVerified`, force the handoff while still in
|
||||||
|
`implement`, and prove the successor sees the complete accepted plan, the
|
||||||
|
verified previous phase, the first unfinished phase, and the current human
|
||||||
|
authority.
|
||||||
|
3. **Finish the remaining ladder.** Manual verification, SHA staleness and
|
||||||
|
reverification, human-decision mismatch, then a real replan with old-plan
|
||||||
|
provenance.
|
||||||
|
4. **Use the capability table above as backend work discovery.** Do not add fake
|
||||||
|
controls. Each disabled action is concrete evidence of a missing capability.
|
||||||
|
5. **Continue the debt slices independently.** Durable worker observations and
|
||||||
|
manual-intervention events are the next evidence gaps. Not automatic
|
||||||
|
maintenance yet.
|
||||||
|
|
||||||
|
## What the tag means
|
||||||
|
|
||||||
|
```text
|
||||||
|
orchestra-release-v1 -> 34f3c28
|
||||||
|
deployed, settled lifecycle plus the truth-detector UI baseline
|
||||||
|
|
||||||
|
later HEADs
|
||||||
|
experimental plan, debt and runtime work that must earn their own
|
||||||
|
release proof
|
||||||
|
```
|
||||||
|
|
||||||
|
Use that distinction. A defect found in experimental work is not a reason to
|
||||||
|
reopen settled architecture.
|
||||||
|
|
||||||
|
## The roadmap the operator set
|
||||||
|
|
||||||
|
```text
|
||||||
|
A. runtime correctness finish the plan-machinery live proof
|
||||||
|
B. evidence and debt durable worker observations, manual-intervention
|
||||||
|
events, then rerun the projection
|
||||||
|
C. operator surface first-class HumanDecision write API,
|
||||||
|
effective project-config read API,
|
||||||
|
session and context health projection
|
||||||
|
D. adapter fix opencode activity and session resolution
|
||||||
|
E. UI wire capabilities as backend support becomes real
|
||||||
|
```
|
||||||
|
|
||||||
|
The framing that ties them together is the thing to keep. The UI says what
|
||||||
|
Orchestra cannot expose or control. The debt ledger says which of those
|
||||||
|
shortcomings repeatedly costs something. The burn-in says which runtime
|
||||||
|
semantics are reliable. Those three decide what gets built next.
|
||||||
|
|
||||||
|
One design question to settle before B's slice two writes any code. The worker
|
||||||
|
ring is bounded and lossy by construction. Ingesting it durably means deciding
|
||||||
|
whether the coordinator stores every observation as an event, or only
|
||||||
|
transitions. Storing every heartbeat's ring would write the same 41-count
|
||||||
|
observation hundreds of times.
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# orchestra ui mockups
|
||||||
|
|
||||||
|
`final/` contains the nine accepted/current screen directions:
|
||||||
|
|
||||||
|
1. dashboard
|
||||||
|
2. task detail
|
||||||
|
3. terminal live pane
|
||||||
|
4. tasks
|
||||||
|
5. decisions
|
||||||
|
6. workers
|
||||||
|
7. projects
|
||||||
|
8. review
|
||||||
|
9. settings
|
||||||
|
|
||||||
|
`iterations/` contains every generated mockup from the design session, including superseded variants.
|
||||||
|
|
||||||
|
`orchestra-ui-spec.md` is the accompanying implementation/design specification.
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"orchestra/internal/domain"
|
||||||
|
"orchestra/internal/federation"
|
||||||
|
"orchestra/internal/herdr"
|
||||||
|
)
|
||||||
|
|
||||||
|
// F62. Run 16: the agent was asked to hand off, never wrote HANDOFF.md,
|
||||||
|
// renewals stopped, and the lease died as ordinary idleness. Waiting is now
|
||||||
|
// bounded: re-ask once, then give the task up with a class that says why.
|
||||||
|
func TestUnansweredHandoffIsRetriedThenGivenUp(t *testing.T) {
|
||||||
|
var nack map[string]any
|
||||||
|
w, backend, _, done := phaseWorker(t, func(rw http.ResponseWriter, r *http.Request) {
|
||||||
|
if strings.HasSuffix(r.URL.Path, "/nack") {
|
||||||
|
_ = json.NewDecoder(r.Body).Decode(&nack)
|
||||||
|
}
|
||||||
|
rw.Write([]byte(`{}`))
|
||||||
|
})
|
||||||
|
defer done()
|
||||||
|
ctx := context.Background()
|
||||||
|
requested := func(ago time.Duration) herdr.Session {
|
||||||
|
s := w.sessions["task"]
|
||||||
|
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
|
||||||
|
s.HandoffRequestedAt = time.Now().UTC().Add(-ago)
|
||||||
|
w.sessions["task"] = s
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// Still inside the answering window: nothing said, nothing given up.
|
||||||
|
if s, gaveUp := w.watchHandoff(ctx, "task", requested(time.Minute)); gaveUp || s.HandoffRetried {
|
||||||
|
t.Fatalf("gave up while still waiting: gaveUp=%v session=%+v", gaveUp, s)
|
||||||
|
}
|
||||||
|
if len(backend.prompts) != 0 {
|
||||||
|
t.Fatalf("re-asked too early: %q", backend.prompts)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Past the retry point: asked again, exactly once.
|
||||||
|
s, gaveUp := w.watchHandoff(ctx, "task", requested(handoffRetryAfter+time.Minute))
|
||||||
|
if gaveUp || !s.HandoffRetried || len(backend.prompts) != 1 {
|
||||||
|
t.Fatalf("retry: gaveUp=%v retried=%v prompts=%q", gaveUp, s.HandoffRetried, backend.prompts)
|
||||||
|
}
|
||||||
|
if _, gaveUp = w.watchHandoff(ctx, "task", s); gaveUp || len(backend.prompts) != 1 {
|
||||||
|
t.Fatalf("re-asked every tick: %q", backend.prompts)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Past the bound: a causal reclaim, and no lease left to renew.
|
||||||
|
s = requested(handoffAnswerTimeout + time.Second)
|
||||||
|
s.HandoffRetried = true
|
||||||
|
w.sessions["task"] = s
|
||||||
|
if _, gaveUp = w.watchHandoff(ctx, "task", s); !gaveUp {
|
||||||
|
t.Fatal("an unanswered handoff waited forever")
|
||||||
|
}
|
||||||
|
if nack["failure_class"] != "handoff_unanswered" {
|
||||||
|
t.Fatalf("nack = %+v", nack)
|
||||||
|
}
|
||||||
|
if detail, _ := nack["last_error"].(string); !strings.Contains(detail, "phase_changed") {
|
||||||
|
t.Fatalf("the reclaim does not name the request: %q", detail)
|
||||||
|
}
|
||||||
|
if _, held := w.leases["task"]; held {
|
||||||
|
t.Fatal("the given-up task kept its lease")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The lease must survive the wait it was asked to make: an idle pane is the
|
||||||
|
// answer Orchestra requested, not evidence of an agent that stopped working.
|
||||||
|
func TestWaitingForAHandoffKeepsTheLease(t *testing.T) {
|
||||||
|
renewals := 0
|
||||||
|
api := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
|
||||||
|
renewals++
|
||||||
|
rw.Write([]byte(`{}`))
|
||||||
|
}))
|
||||||
|
defer api.Close()
|
||||||
|
backend := &recordingBackend{status: "idle", progress: "same screen"}
|
||||||
|
w := &worker{
|
||||||
|
api: federation.Client{BaseURL: api.URL, WorkerID: "h", Token: "t"},
|
||||||
|
backend: backend,
|
||||||
|
harness: "claude",
|
||||||
|
sessions: map[string]herdr.Session{"task": {PaneID: "pane", HandoffRequested: true, HandoffRequestedAt: time.Now().UTC()}},
|
||||||
|
leases: map[string]lease{"task": {Epoch: "e", Version: 1, Until: time.Now(), ProgressSHA: domain.Hash([]byte("same screen"))}},
|
||||||
|
quarantined: map[string]bool{},
|
||||||
|
statePath: filepath.Join(t.TempDir(), "state.json"),
|
||||||
|
}
|
||||||
|
w.renewLeases(context.Background())
|
||||||
|
if renewals != 1 {
|
||||||
|
t.Fatalf("a lease waiting on a requested handoff renewed %d times, want 1", renewals)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Past the bound the exemption stops: watchHandoff has given the task up
|
||||||
|
// by then, and nothing keeps an unanswered request alive.
|
||||||
|
s := w.sessions["task"]
|
||||||
|
s.HandoffRequestedAt = time.Now().UTC().Add(-handoffAnswerTimeout - time.Second)
|
||||||
|
w.sessions["task"] = s
|
||||||
|
l := w.leases["task"]
|
||||||
|
l.Until = time.Now()
|
||||||
|
w.leases["task"] = l
|
||||||
|
w.renewLeases(context.Background())
|
||||||
|
if renewals != 1 {
|
||||||
|
t.Fatalf("the exemption outlived its bound: renewals=%d", renewals)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -174,6 +174,7 @@ func releaseBackoff(attempts int) time.Duration {
|
|||||||
}
|
}
|
||||||
return d
|
return d
|
||||||
}
|
}
|
||||||
|
|
||||||
type projectConfig struct {
|
type projectConfig struct {
|
||||||
Repo string `json:"repo"`
|
Repo string `json:"repo"`
|
||||||
Root string `json:"worktree_root"`
|
Root string `json:"worktree_root"`
|
||||||
@@ -673,6 +674,13 @@ func (w *worker) releaseReady(ctx context.Context) {
|
|||||||
w.advanceRelease(ctx, id, s)
|
w.advanceRelease(ctx, id, s)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
if s.HandoffRequested {
|
||||||
|
next, gaveUp := w.watchHandoff(ctx, id, s)
|
||||||
|
if gaveUp {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
s = next
|
||||||
|
}
|
||||||
w.rotationTick(ctx, id, s)
|
w.rotationTick(ctx, id, s)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -772,7 +780,7 @@ func (w *worker) rotationTick(ctx context.Context, id string, s herdr.Session) {
|
|||||||
w.recordError(fmt.Errorf("rotation %s threshold prompt: %w", id, err))
|
w.recordError(fmt.Errorf("rotation %s threshold prompt: %w", id, err))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
s.HandoffRequested, s.HandoffReason = true, d.Reason
|
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, d.Reason, time.Now().UTC()
|
||||||
w.sessions[id] = s
|
w.sessions[id] = s
|
||||||
_ = w.save()
|
_ = w.save()
|
||||||
}
|
}
|
||||||
@@ -1166,6 +1174,70 @@ func (w *worker) submit(ctx context.Context, id string, s herdr.Session, e compl
|
|||||||
// the directory also holds for a worktree that has no inner .gitignore.
|
// the directory also holds for a worktree that has no inner .gitignore.
|
||||||
const stageExclude = ":!.orchestra"
|
const stageExclude = ":!.orchestra"
|
||||||
|
|
||||||
|
// F62: a requested handoff nobody answers was invisible. Renewals stopped,
|
||||||
|
// the lease expired, and the task lost an attempt with nothing on record
|
||||||
|
// saying a handoff had ever been asked for — worker health showed only "agent
|
||||||
|
// status idle and pane unchanged", 34 times in run 16.
|
||||||
|
const (
|
||||||
|
handoffRetryAfter = 4 * time.Minute
|
||||||
|
handoffAnswerTimeout = 10 * time.Minute
|
||||||
|
)
|
||||||
|
|
||||||
|
// watchHandoff bounds the wait for an agent's handoff answer: re-send the
|
||||||
|
// request once, then give the task up with a class that names the cause. It
|
||||||
|
// returns the session to keep using and whether the task was given up.
|
||||||
|
func (w *worker) watchHandoff(ctx context.Context, id string, s herdr.Session) (herdr.Session, bool) {
|
||||||
|
if s.HandoffRequestedAt.IsZero() {
|
||||||
|
// A session persisted before the stamp existed, or requested by a path
|
||||||
|
// that does not set it. Start the clock now rather than time out a
|
||||||
|
// request retroactively.
|
||||||
|
s.HandoffRequestedAt = time.Now().UTC()
|
||||||
|
w.sessions[id] = s
|
||||||
|
_ = w.save()
|
||||||
|
return s, false
|
||||||
|
}
|
||||||
|
waited := time.Since(s.HandoffRequestedAt)
|
||||||
|
if waited < handoffRetryAfter {
|
||||||
|
return s, false
|
||||||
|
}
|
||||||
|
if waited < handoffAnswerTimeout {
|
||||||
|
if s.HandoffRetried || w.executionBackend() == nil {
|
||||||
|
return s, false
|
||||||
|
}
|
||||||
|
a := herdr.CLIAdapter{Backend: w.executionBackend(), Harness: w.harness}
|
||||||
|
var err error
|
||||||
|
if s.HandoffReason != "" {
|
||||||
|
err = a.RequestHandoffReason(ctx, s, s.HandoffReason, nil)
|
||||||
|
} else {
|
||||||
|
err = a.RequestHandoff(ctx, s)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
w.recordError(fmt.Errorf("handoff %s re-request: %w", id, err))
|
||||||
|
return s, false
|
||||||
|
}
|
||||||
|
s.HandoffRetried = true
|
||||||
|
w.sessions[id] = s
|
||||||
|
_ = w.save()
|
||||||
|
w.recordError(fmt.Errorf("handoff %s (%s) unanswered for %s: request re-sent", id, s.HandoffReason, waited.Round(time.Second)))
|
||||||
|
return s, false
|
||||||
|
}
|
||||||
|
l, ok := w.leases[id]
|
||||||
|
if !ok {
|
||||||
|
return s, false
|
||||||
|
}
|
||||||
|
detail := fmt.Sprintf("handoff requested (%s) and unanswered for %s", s.HandoffReason, waited.Round(time.Second))
|
||||||
|
if err := w.api.NackStart(ctx, id, l.Epoch, l.Version, "handoff_unanswered", detail, w.sessionEvidence(ctx, id, s)); err != nil {
|
||||||
|
w.recordError(fmt.Errorf("handoff timeout %s: %w", id, err))
|
||||||
|
return s, false
|
||||||
|
}
|
||||||
|
// The coordinator answers with TaskReleased; its replay quarantines the
|
||||||
|
// pane. Drop the lease here so nothing renews it in the meantime.
|
||||||
|
delete(w.leases, id)
|
||||||
|
_ = w.save()
|
||||||
|
w.recordError(errors.New(detail))
|
||||||
|
return s, true
|
||||||
|
}
|
||||||
|
|
||||||
func (w *worker) renewLeases(ctx context.Context) {
|
func (w *worker) renewLeases(ctx context.Context) {
|
||||||
if w.executionBackend() == nil {
|
if w.executionBackend() == nil {
|
||||||
return
|
return
|
||||||
@@ -1199,6 +1271,11 @@ func (w *worker) renewLeases(ctx context.Context) {
|
|||||||
case l.ProgressSHA == "":
|
case l.ProgressSHA == "":
|
||||||
// First renewal has no baseline to compare against. Record one and
|
// First renewal has no baseline to compare against. Record one and
|
||||||
// allow this renewal; the next one must show real movement.
|
// allow this renewal; the next one must show real movement.
|
||||||
|
case s.HandoffRequested && !s.HandoffRequestedAt.IsZero() && time.Since(s.HandoffRequestedAt) < handoffAnswerTimeout:
|
||||||
|
// Orchestra told this agent to stop and write its handoff. A quiet
|
||||||
|
// pane is the answer it was asked for, so the lease is held while
|
||||||
|
// the wait is explicitly bounded (F62). Past the bound the case
|
||||||
|
// stops matching and watchHandoff has already given the task up.
|
||||||
default:
|
default:
|
||||||
w.recordError(fmt.Errorf("lease %s not renewed: agent status %s and pane unchanged since the last renewal", taskID, status))
|
w.recordError(fmt.Errorf("lease %s not renewed: agent status %s and pane unchanged since the last renewal", taskID, status))
|
||||||
continue
|
continue
|
||||||
@@ -1932,7 +2009,7 @@ func (w *worker) federatedTurn(ctx context.Context, id string, a herdr.Adapter,
|
|||||||
w.recordError(fmt.Errorf("reconcile failure handoff %s: %w", id, err))
|
w.recordError(fmt.Errorf("reconcile failure handoff %s: %w", id, err))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
s.HandoffRequested, s.HandoffReason = true, "reconcile_failure"
|
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "reconcile_failure", time.Now().UTC()
|
||||||
w.sessions[id] = s
|
w.sessions[id] = s
|
||||||
_ = w.save()
|
_ = w.save()
|
||||||
return
|
return
|
||||||
@@ -2009,7 +2086,7 @@ func (w *worker) rotateForPhase(ctx context.Context, id string, a herdr.Adapter,
|
|||||||
w.recordError(fmt.Errorf("phase rotation %s: %w", id, err))
|
w.recordError(fmt.Errorf("phase rotation %s: %w", id, err))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
|
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "phase_changed", time.Now().UTC()
|
||||||
w.sessions[id] = s
|
w.sessions[id] = s
|
||||||
_ = w.save()
|
_ = w.save()
|
||||||
log.Printf("phase changed for %s: session rotating", id)
|
log.Printf("phase changed for %s: session rotating", id)
|
||||||
|
|||||||
@@ -1589,6 +1589,12 @@ func main() {
|
|||||||
case "invalid_handoff":
|
case "invalid_handoff":
|
||||||
typ = "TaskBlocked"
|
typ = "TaskBlocked"
|
||||||
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonHandoffValidation), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_nacked", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonHandoffValidation), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_nacked", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
||||||
|
case "handoff_unanswered":
|
||||||
|
// F62: not a launch failure. The agent was asked to hand off
|
||||||
|
// and never did, so the reclaim says exactly that instead of
|
||||||
|
// arriving as an ordinary idle expiry.
|
||||||
|
typ = "TaskReleased"
|
||||||
|
p, _ = json.Marshal(map[string]any{"reason": "handoff_unanswered", "failure_class": b.FailureClass, "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "handoff_unanswered", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
||||||
case "launch_uncertain":
|
case "launch_uncertain":
|
||||||
typ = "TaskNeedsAttention"
|
typ = "TaskNeedsAttention"
|
||||||
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonLeaseFailure), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_uncertain", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonLeaseFailure), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_uncertain", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
||||||
@@ -1825,6 +1831,25 @@ func main() {
|
|||||||
coordinator.ReconcileFailureHandoff = v
|
coordinator.ReconcileFailureHandoff = v
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// The third reconciliation point, and the same reasoning the submitted
|
||||||
|
// work loop above already uses: a blocked task cannot be leased, so a
|
||||||
|
// pre-lease hook can never see the reply that should unblock it (F64).
|
||||||
|
// Orchestra stopped and asked the human a question; nothing else here
|
||||||
|
// will go and read the answer.
|
||||||
|
go func() {
|
||||||
|
ticker := time.NewTicker(time.Minute)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for range ticker.C {
|
||||||
|
for _, t := range s.Tasks() {
|
||||||
|
if t.State != domain.StateBlocked || !domain.BlockReasonAwaitsReply(t.BlockReason) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := reconciler.Reconcile(context.Background(), t.ID); err != nil {
|
||||||
|
log.Printf("reconcile blocked task %s: %v", t.ID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
}
|
}
|
||||||
if path := os.Getenv("ORCHESTRA_JSONL"); path != "" {
|
if path := os.Getenv("ORCHESTRA_JSONL"); path != "" {
|
||||||
sup := &provider.Supervisor{Name: "jsonl", Run: func(ctx context.Context) error {
|
sup := &provider.Supervisor{Name: "jsonl", Run: func(ctx context.Context) error {
|
||||||
|
|||||||
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
@@ -251,7 +251,7 @@ func DebtClassForBlockReason(r BlockReason) (DebtClass, bool) {
|
|||||||
// classes a worker actually emits are listed; an unknown one is not guessed at.
|
// classes a worker actually emits are listed; an unknown one is not guessed at.
|
||||||
func DebtClassForFailureClass(f string) (DebtClass, bool) {
|
func DebtClassForFailureClass(f string) (DebtClass, bool) {
|
||||||
switch f {
|
switch f {
|
||||||
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired":
|
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired", "handoff_unanswered":
|
||||||
return DebtOperational, true
|
return DebtOperational, true
|
||||||
case "invalid_handoff":
|
case "invalid_handoff":
|
||||||
return DebtCorrectness, true
|
return DebtCorrectness, true
|
||||||
|
|||||||
@@ -82,6 +82,23 @@ const (
|
|||||||
BlockReasonUnknown BlockReason = "unknown"
|
BlockReasonUnknown BlockReason = "unknown"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// BlockReasonAwaitsReply reports whether this stop is a question waiting on a
|
||||||
|
// human, rather than a fault or a budget. Every reason listed here resumes on
|
||||||
|
// a reply, so both the loop that goes and reads replies and the loop that
|
||||||
|
// returns answered tasks to the queue must agree on the set. They did not:
|
||||||
|
// plan_mismatch blocked for an answer that nothing ever went to fetch, and
|
||||||
|
// nothing would have resumed it if it had (F64).
|
||||||
|
//
|
||||||
|
// operator_required is deliberately absent. A task that has spent its question
|
||||||
|
// budget continues when an operator says so, not when someone replies.
|
||||||
|
func BlockReasonAwaitsReply(r BlockReason) bool {
|
||||||
|
switch r {
|
||||||
|
case BlockReasonHumanDecision, BlockReasonTrajectoryGate, BlockReasonPlanMismatch:
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
func (r BlockReason) Valid() bool {
|
func (r BlockReason) Valid() bool {
|
||||||
switch r {
|
switch r {
|
||||||
case BlockReasonLeaseFailure, BlockReasonWorkerOffline, BlockReasonLeaseExpired,
|
case BlockReasonLeaseFailure, BlockReasonWorkerOffline, BlockReasonLeaseExpired,
|
||||||
@@ -168,19 +185,19 @@ type Task struct {
|
|||||||
QualityGate string `json:"quality_gate,omitempty"`
|
QualityGate string `json:"quality_gate,omitempty"`
|
||||||
// Block evidence is projected from TaskBlocked so terminal records remain
|
// Block evidence is projected from TaskBlocked so terminal records remain
|
||||||
// diagnosable after the live coordinator mapping is gone.
|
// diagnosable after the live coordinator mapping is gone.
|
||||||
Blocker string `json:"blocker,omitempty"`
|
Blocker string `json:"blocker,omitempty"`
|
||||||
BlockReason BlockReason `json:"block_reason,omitempty"`
|
BlockReason BlockReason `json:"block_reason,omitempty"`
|
||||||
BlockedAt time.Time `json:"blocked_at,omitempty"`
|
BlockedAt time.Time `json:"blocked_at,omitempty"`
|
||||||
LastPaneID string `json:"last_pane_id,omitempty"`
|
LastPaneID string `json:"last_pane_id,omitempty"`
|
||||||
LastHarness string `json:"last_harness_id,omitempty"`
|
LastHarness string `json:"last_harness_id,omitempty"`
|
||||||
// LastLeaseEpoch is the fencing token of the lease that most recently
|
// LastLeaseEpoch is the fencing token of the lease that most recently
|
||||||
// ended. A worker can push its release anchor and only then discover the
|
// ended. A worker can push its release anchor and only then discover the
|
||||||
// lease expired; the finished work is durable in git but the commit can
|
// lease expired; the finished work is durable in git but the commit can
|
||||||
// never land. Retaining the epoch lets exactly that owner still commit
|
// never land. Retaining the epoch lets exactly that owner still commit
|
||||||
// while the task sits unleased.
|
// while the task sits unleased.
|
||||||
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
|
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
|
||||||
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
|
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
|
||||||
LastSession SessionEvidence `json:"last_session,omitempty"`
|
LastSession SessionEvidence `json:"last_session,omitempty"`
|
||||||
// Recovery state is part of the durable projection, never process-local
|
// Recovery state is part of the durable projection, never process-local
|
||||||
// router memory. This makes retry and operator diagnostics survive a
|
// router memory. This makes retry and operator diagnostics survive a
|
||||||
// coordinator restart.
|
// coordinator restart.
|
||||||
@@ -217,7 +234,7 @@ type Task struct {
|
|||||||
// current one, oldest first. A superseded plan stays queryable: the
|
// current one, oldest first. A superseded plan stays queryable: the
|
||||||
// verification recorded against it is provenance, not garbage.
|
// verification recorded against it is provenance, not garbage.
|
||||||
PlanHistory []string `json:"plan_history,omitempty"`
|
PlanHistory []string `json:"plan_history,omitempty"`
|
||||||
LastError string `json:"last_error,omitempty"`
|
LastError string `json:"last_error,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ReviewRef binds a sealed review artifact to one commit.
|
// ReviewRef binds a sealed review artifact to one commit.
|
||||||
|
|||||||
@@ -55,6 +55,13 @@ type PlanPhaseRecord struct {
|
|||||||
// EvidenceRef is the CAS ref of the captured command output.
|
// EvidenceRef is the CAS ref of the captured command output.
|
||||||
EvidenceRef string `json:"evidence_ref,omitempty"`
|
EvidenceRef string `json:"evidence_ref,omitempty"`
|
||||||
At time.Time `json:"at"`
|
At time.Time `json:"at"`
|
||||||
|
// ManualAtSHA is the tree a human was actually looking at when they signed
|
||||||
|
// this phase off. A manual check on most projects is a human reading
|
||||||
|
// output, so a sign-off establishes something about one tree and nothing
|
||||||
|
// about the next one (F63). Rerunning the automated half re-establishes it
|
||||||
|
// at the new commit; the manual half has to be given again, and this is
|
||||||
|
// what makes the difference visible instead of assumed.
|
||||||
|
ManualAtSHA string `json:"manual_at_sha,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stale reports whether the tree has moved since this phase was verified. A
|
// Stale reports whether the tree has moved since this phase was verified. A
|
||||||
|
|||||||
@@ -174,6 +174,14 @@ type Session struct {
|
|||||||
// its §6.1 handoff (HandoffFile) — avoids re-sending the same prompt
|
// its §6.1 handoff (HandoffFile) — avoids re-sending the same prompt
|
||||||
// every tick while Release keeps waiting for the file to appear.
|
// every tick while Release keeps waiting for the file to appear.
|
||||||
HandoffRequested bool `json:"handoff_requested,omitempty"`
|
HandoffRequested bool `json:"handoff_requested,omitempty"`
|
||||||
|
// HandoffRequestedAt stamps that prompt. A request nobody answers used to
|
||||||
|
// end as an ordinary idle expiry, indistinguishable from an agent that
|
||||||
|
// never started (F62); the stamp is what makes the wait bounded and the
|
||||||
|
// giving-up causal.
|
||||||
|
HandoffRequestedAt time.Time `json:"handoff_requested_at,omitempty"`
|
||||||
|
// HandoffRetried records that the request was re-sent once, so a session
|
||||||
|
// waiting on an answer is not re-prompted every tick.
|
||||||
|
HandoffRetried bool `json:"handoff_retried,omitempty"`
|
||||||
// HandoffReason is selected by the coordinator when it asks for the
|
// HandoffReason is selected by the coordinator when it asks for the
|
||||||
// semantic report. The checkout worker, rather than the harness, copies
|
// semantic report. The checkout worker, rather than the harness, copies
|
||||||
// it into the canonical handoff it seals at release time.
|
// it into the canonical handoff it seals at release time.
|
||||||
|
|||||||
@@ -107,11 +107,9 @@ func ResumeAnsweredBlockers(s *store.Store) ([]domain.Event, error) {
|
|||||||
if t.State != domain.StateBlocked {
|
if t.State != domain.StateBlocked {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
switch t.BlockReason {
|
// operator_required is deliberately not resumed by a reply. An
|
||||||
case domain.BlockReasonHumanDecision, domain.BlockReasonTrajectoryGate:
|
// operator decides when a task that spent its budget continues.
|
||||||
default:
|
if !domain.BlockReasonAwaitsReply(t.BlockReason) {
|
||||||
// operator_required is deliberately not resumed by a reply. An
|
|
||||||
// operator decides when a task that spent its budget continues.
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if !blockerAnswered(s, t.ID, t.BlockReason) {
|
if !blockerAnswered(s, t.ID, t.BlockReason) {
|
||||||
|
|||||||
@@ -215,7 +215,16 @@ func TestHumanAnswerResumesTheSamePlanWithoutResealing(t *testing.T) {
|
|||||||
if !PlanMismatchAnswered(s, id) {
|
if !PlanMismatchAnswered(s, id) {
|
||||||
t.Fatal("the human answered and the task is still waiting")
|
t.Fatal("the human answered and the task is still waiting")
|
||||||
}
|
}
|
||||||
|
// F64: asserting the predicate is not asserting the resume. This test
|
||||||
|
// passed for as long as the predicate had no caller, while a task blocked
|
||||||
|
// on a mismatch stayed blocked forever however the human replied.
|
||||||
|
if events, err := ResumeAnsweredBlockers(s); err != nil || len(events) != 1 {
|
||||||
|
t.Fatalf("an answered mismatch did not return to the queue: events=%v err=%v", events, err)
|
||||||
|
}
|
||||||
after, _ := s.Task(id)
|
after, _ := s.Task(id)
|
||||||
|
if after.State != domain.StateQueued {
|
||||||
|
t.Fatalf("state = %s after the human answered, want queued", after.State)
|
||||||
|
}
|
||||||
if after.PlanRef != planRef {
|
if after.PlanRef != planRef {
|
||||||
t.Fatal("answering the question replaced the plan")
|
t.Fatal("answering the question replaced the plan")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -124,7 +124,12 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
|
|||||||
// was established.
|
// was established.
|
||||||
record.Status = domain.PlanPhaseAwaitingManual
|
record.Status = domain.PlanPhaseAwaitingManual
|
||||||
}
|
}
|
||||||
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID) {
|
// Carry the confirmed tree forward as provenance. Without it a second
|
||||||
|
// rerun would compare against nothing and re-inherit the sign-off.
|
||||||
|
if prior, ok := t.PlanPhase(phaseID); ok {
|
||||||
|
record.ManualAtSHA = prior.ManualAtSHA
|
||||||
|
}
|
||||||
|
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID, record) {
|
||||||
record.Status = domain.PlanPhaseVerified
|
record.Status = domain.PlanPhaseVerified
|
||||||
}
|
}
|
||||||
if ref, err := s.PutArtifact(verificationEvidence(runs)); err == nil {
|
if ref, err := s.PutArtifact(verificationEvidence(runs)); err == nil {
|
||||||
@@ -135,7 +140,7 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
|
|||||||
payload := map[string]any{
|
payload := map[string]any{
|
||||||
"plan_ref": record.PlanRef, "phase_id": record.PhaseID, "status": string(record.Status),
|
"plan_ref": record.PlanRef, "phase_id": record.PhaseID, "status": string(record.Status),
|
||||||
"commands": record.Commands, "exit_codes": record.ExitCodes, "at_sha": record.AtSHA,
|
"commands": record.Commands, "exit_codes": record.ExitCodes, "at_sha": record.AtSHA,
|
||||||
"evidence_ref": record.EvidenceRef, "at": record.At,
|
"evidence_ref": record.EvidenceRef, "at": record.At, "manual_at_sha": record.ManualAtSHA,
|
||||||
}
|
}
|
||||||
if t.Lease != nil {
|
if t.Lease != nil {
|
||||||
payload["harness_id"], payload["lease_epoch"] = t.Lease.HarnessID, t.Lease.Epoch
|
payload["harness_id"], payload["lease_epoch"] = t.Lease.HarnessID, t.Lease.Epoch
|
||||||
@@ -149,9 +154,20 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
|
|||||||
}
|
}
|
||||||
|
|
||||||
// manuallySignedOff reports whether a human has already approved this exact
|
// manuallySignedOff reports whether a human has already approved this exact
|
||||||
// phase of this exact plan. The subject carries both, so a later "looks good"
|
// phase of this exact plan, against the tree this run examined. The subject
|
||||||
// on an unrelated thread cannot satisfy a gate nobody was discussing.
|
// carries plan and phase, so a later "looks good" on an unrelated thread
|
||||||
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string) bool {
|
// cannot satisfy a gate nobody was discussing.
|
||||||
|
//
|
||||||
|
// The tree matters as much as the subject (F63). A sign-off is a human saying
|
||||||
|
// they read what this code prints; an edit afterwards can change exactly that.
|
||||||
|
// A record whose ManualAtSHA names a different commit is therefore not signed
|
||||||
|
// off, and waits for the human again. A sign-off given before any run has no
|
||||||
|
// confirmed tree to compare against and still counts, which keeps the ordinary
|
||||||
|
// ordering unchanged.
|
||||||
|
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string, record domain.PlanPhaseRecord) bool {
|
||||||
|
if record.ManualAtSHA != "" && record.ManualAtSHA != record.AtSHA {
|
||||||
|
return false
|
||||||
|
}
|
||||||
intent, err := s.EffectiveIntent(t.ID)
|
intent, err := s.EffectiveIntent(t.ID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false
|
return false
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ import (
|
|||||||
|
|
||||||
const shaOne = "1111111111111111111111111111111111111111"
|
const shaOne = "1111111111111111111111111111111111111111"
|
||||||
const shaTwo = "2222222222222222222222222222222222222222"
|
const shaTwo = "2222222222222222222222222222222222222222"
|
||||||
|
const shaThree = "3333333333333333333333333333333333333333"
|
||||||
|
|
||||||
func planProject() registry.Project {
|
func planProject() registry.Project {
|
||||||
p := registry.Project{
|
p := registry.Project{
|
||||||
@@ -251,6 +252,14 @@ func TestLegacyPlanIsExplicitlyNonProgressable(t *testing.T) {
|
|||||||
// signOff records a human decision bound to one phase of one plan, which is
|
// signOff records a human decision bound to one phase of one plan, which is
|
||||||
// the only thing that satisfies a manual verification gate.
|
// the only thing that satisfies a manual verification gate.
|
||||||
func signOff(t *testing.T, s *store.Store, taskID, subject string) {
|
func signOff(t *testing.T, s *store.Store, taskID, subject string) {
|
||||||
|
t.Helper()
|
||||||
|
signOffFrom(t, s, taskID, subject, "signoff-"+subject)
|
||||||
|
}
|
||||||
|
|
||||||
|
// signOffFrom names the comment the sign-off came from. Two sign-offs on one
|
||||||
|
// subject are a real sequence once a rerun sends a phase back to the human,
|
||||||
|
// and provenance is unique per comment.
|
||||||
|
func signOffFrom(t *testing.T, s *store.Store, taskID, subject, externalID string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
task, _ := s.Task(taskID)
|
task, _ := s.Task(taskID)
|
||||||
if err := s.Append(domain.Event{
|
if err := s.Append(domain.Event{
|
||||||
@@ -259,9 +268,60 @@ func signOff(t *testing.T, s *store.Store, taskID, subject string) {
|
|||||||
Payload: mustJSONBytes(t, map[string]any{
|
Payload: mustJSONBytes(t, map[string]any{
|
||||||
"decision_id": domain.NewID(), "kind": "answer", "subject": subject,
|
"decision_id": domain.NewID(), "kind": "answer", "subject": subject,
|
||||||
"value": "manual steps confirmed",
|
"value": "manual steps confirmed",
|
||||||
"source": map[string]any{"provider": "gitea", "external_id": "signoff-" + subject},
|
"source": map[string]any{"provider": "gitea", "external_id": externalID},
|
||||||
}),
|
}),
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("sign off: %v", err)
|
t.Fatalf("sign off: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// F63, found live on run 19. A manual sign-off says a human read what this
|
||||||
|
// code prints. An edit afterwards can change exactly that, so rerunning the
|
||||||
|
// automated half at a new commit must not carry the human half with it.
|
||||||
|
func TestASignOffDoesNotSurviveTheTreeItWasGivenAgainst(t *testing.T) {
|
||||||
|
s, project, id := planWith(t, twoPhasePlan)
|
||||||
|
run := []VerificationRun{{Command: []string{"go", "test", "./internal/..."}, ExitCode: 0}}
|
||||||
|
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaOne, run); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
task, _ := s.Task(id)
|
||||||
|
signOff(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"))
|
||||||
|
task, _ = s.Task(id)
|
||||||
|
rec, _ := task.PlanPhase("phase-2")
|
||||||
|
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaOne {
|
||||||
|
t.Fatalf("sign-off did not bind to the tree it read: %+v", rec)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The tree moves and the phase is re-verified. The commands pass again;
|
||||||
|
// the human has not seen the new output.
|
||||||
|
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaTwo, run); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
task, _ = s.Task(id)
|
||||||
|
rec, _ = task.PlanPhase("phase-2")
|
||||||
|
if rec.Status != domain.PlanPhaseAwaitingManual {
|
||||||
|
t.Fatalf("status = %q at a tree the human never saw, want awaiting_manual_verification", rec.Status)
|
||||||
|
}
|
||||||
|
if rec.ManualAtSHA != shaOne {
|
||||||
|
t.Fatalf("the confirmed tree was lost: %+v", rec)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A second rerun must not re-inherit it either, which is what carrying
|
||||||
|
// ManualAtSHA forward is for.
|
||||||
|
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaThree, run); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
task, _ = s.Task(id)
|
||||||
|
rec, _ = task.PlanPhase("phase-2")
|
||||||
|
if rec.Status != domain.PlanPhaseAwaitingManual {
|
||||||
|
t.Fatalf("a second rerun re-inherited the sign-off: %q", rec.Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Signing off again, on the tree that is now current, verifies it.
|
||||||
|
signOffFrom(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"), "signoff-second")
|
||||||
|
task, _ = s.Task(id)
|
||||||
|
rec, _ = task.PlanPhase("phase-2")
|
||||||
|
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaThree {
|
||||||
|
t.Fatalf("a fresh sign-off did not verify the current tree: %+v", rec)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -272,6 +272,9 @@ func (s *Store) apply(e domain.Event) error {
|
|||||||
}
|
}
|
||||||
if p.Subject == domain.PlanPhaseSubject(rec.PlanRef, rec.PhaseID) {
|
if p.Subject == domain.PlanPhaseSubject(rec.PlanRef, rec.PhaseID) {
|
||||||
t.PlanProgress.Phases[i].Status = domain.PlanPhaseVerified
|
t.PlanProgress.Phases[i].Status = domain.PlanPhaseVerified
|
||||||
|
// Record which tree the sign-off was about, so a later run
|
||||||
|
// at a different commit cannot inherit it (F63).
|
||||||
|
t.PlanProgress.Phases[i].ManualAtSHA = rec.AtSHA
|
||||||
t.Version = e.Version
|
t.Version = e.Version
|
||||||
s.replaceTask(e.TaskID, t)
|
s.replaceTask(e.TaskID, t)
|
||||||
}
|
}
|
||||||
|
|||||||
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
@@ -0,0 +1,635 @@
|
|||||||
|
# orchestra web control ui
|
||||||
|
|
||||||
|
status: mockup specification
|
||||||
|
app: orchestra
|
||||||
|
shared design system: ethos
|
||||||
|
accent: signal violet `#8F7AE5`
|
||||||
|
motif: routing fork
|
||||||
|
primary operator: single infrastructure operator
|
||||||
|
|
||||||
|
## 1. purpose
|
||||||
|
|
||||||
|
orchestra's web ui is an operator console for understanding and steering agent work without reconstructing state from logs, tmux sessions, gitea, or the event stream.
|
||||||
|
|
||||||
|
it has three jobs:
|
||||||
|
|
||||||
|
1. show what needs the operator now.
|
||||||
|
2. show what orchestra believes and what an agent is actually doing.
|
||||||
|
3. expose intervention and forensic detail without making the default screens noisy.
|
||||||
|
|
||||||
|
it is not a generic project-management product, an analytics dashboard, or a chat-first interface.
|
||||||
|
|
||||||
|
## 2. ethos application
|
||||||
|
|
||||||
|
orchestra inherits the ethos shell and neutral system unchanged.
|
||||||
|
|
||||||
|
### fingerprint
|
||||||
|
|
||||||
|
- accent: signal violet `#8F7AE5`
|
||||||
|
- motif: routing fork
|
||||||
|
- accent use: active navigation, focus, selected phase, primary operator action, routing/transition marks
|
||||||
|
- motif use: app mark, phase transitions, routing state, empty states, one hero/detail moment
|
||||||
|
|
||||||
|
suggested token block:
|
||||||
|
|
||||||
|
```css
|
||||||
|
[data-app="orchestra"] {
|
||||||
|
--accent: #8F7AE5;
|
||||||
|
--accent-hi: #A291EC;
|
||||||
|
--accent-dim: rgba(143, 122, 229, 0.14);
|
||||||
|
--accent-line: rgba(143, 122, 229, 0.32);
|
||||||
|
--accent-glow: rgba(143, 122, 229, 0.22);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### type
|
||||||
|
|
||||||
|
- geist sans: labels, task names, explanations, buttons, headings, operator-authored prose
|
||||||
|
- geist mono: task ids, epochs, worker ids, pane ids, timestamps, percentages, context counts, revisions, paths, refs, sha values, lease durations
|
||||||
|
- machine values must never silently use sans for visual convenience
|
||||||
|
|
||||||
|
### surfaces
|
||||||
|
|
||||||
|
- page room: `--bg-0`
|
||||||
|
- normal panel: `--bg-1`
|
||||||
|
- selected/raised content: `--bg-2`
|
||||||
|
- hover: `--bg-3`
|
||||||
|
- borders: 1px `--line`, `--line-hi` only where selection or risk needs stronger separation
|
||||||
|
- no blur or glass
|
||||||
|
- accent never washes a whole content panel
|
||||||
|
|
||||||
|
### motion
|
||||||
|
|
||||||
|
all ui transitions use ethos mechanical motion: 130–170 ms, `cubic-bezier(0.2, 0, 0, 1)`, no spring or bounce.
|
||||||
|
|
||||||
|
## 3. shared shell
|
||||||
|
|
||||||
|
### desktop
|
||||||
|
|
||||||
|
- 64px vertical icon rail
|
||||||
|
- 56px top bar
|
||||||
|
- main content fills remaining viewport
|
||||||
|
- global `cmd+k` command/search surface in the top bar
|
||||||
|
- right side of top bar shows only high-value machine state: orchestra revision, live state, current time
|
||||||
|
|
||||||
|
primary navigation:
|
||||||
|
|
||||||
|
1. dashboard
|
||||||
|
2. tasks
|
||||||
|
3. decisions
|
||||||
|
4. workers
|
||||||
|
5. projects
|
||||||
|
6. review
|
||||||
|
7. settings
|
||||||
|
|
||||||
|
terminal is entered from a task or worker and does not need permanent primary navigation.
|
||||||
|
|
||||||
|
### mobile
|
||||||
|
|
||||||
|
- rail becomes bottom navigation with no more than five primary items
|
||||||
|
- secondary pages move behind task/project drill-down or header menus
|
||||||
|
- machine readouts relocate into detail screens rather than disappearing
|
||||||
|
- terminal remains a dedicated full-screen view
|
||||||
|
|
||||||
|
## 4. information hierarchy
|
||||||
|
|
||||||
|
orchestra should visually distinguish five classes of information.
|
||||||
|
|
||||||
|
| class | meaning | treatment |
|
||||||
|
|---|---|---|
|
||||||
|
| operator authority | human decision, correction, constraint | high prominence, accent signal |
|
||||||
|
| verified orchestra state | lifecycle, lease, phase, review, refs | neutral surfaces, machine values in mono |
|
||||||
|
| agent claims | proposed plan, handoff text, self-reported risk | visibly labeled as agent-supplied |
|
||||||
|
| historical/superseded | stale plan, previous handoff, old decision | muted, never visually dominant |
|
||||||
|
| fault/attention | blocked, refused, stale, mismatch | status color + explicit reason |
|
||||||
|
|
||||||
|
raw events and implementation details are drill-down evidence, not default-page content.
|
||||||
|
|
||||||
|
## 5. dashboard
|
||||||
|
|
||||||
|
mockup: `final/01-dashboard.png`
|
||||||
|
|
||||||
|
purpose: answer "what needs me, what is running, and is the system healthy?"
|
||||||
|
|
||||||
|
### layout
|
||||||
|
|
||||||
|
three primary regions only:
|
||||||
|
|
||||||
|
1. **needs your attention**
|
||||||
|
- trajectory gate
|
||||||
|
- blocking human decision
|
||||||
|
- pr awaiting review
|
||||||
|
- closed-without-merge/operator-required state
|
||||||
|
- one line explaining why each item needs the operator
|
||||||
|
|
||||||
|
2. **running tasks**
|
||||||
|
- task name/id
|
||||||
|
- current phase
|
||||||
|
- worker/harness
|
||||||
|
- last real progress
|
||||||
|
- context occupancy
|
||||||
|
- lease timing
|
||||||
|
- compact routing-fork phase indicator
|
||||||
|
|
||||||
|
3. **system state and capacity**
|
||||||
|
- workers online/available
|
||||||
|
- quota status
|
||||||
|
- active leases
|
||||||
|
- router queue/rejection state
|
||||||
|
- source reconciliation
|
||||||
|
- herdr health
|
||||||
|
|
||||||
|
### rules
|
||||||
|
|
||||||
|
- no disk, cpu, ram, or generic host-monitor metrics unless they directly block orchestration
|
||||||
|
- no vanity charts
|
||||||
|
- health should show actual counts/times, not vague "running" indicators
|
||||||
|
- dashboard actions should navigate to the relevant task/review/worker rather than becoming an operator-control surface itself
|
||||||
|
|
||||||
|
## 6. tasks list
|
||||||
|
|
||||||
|
mockup: `final/04-tasks.png`
|
||||||
|
|
||||||
|
purpose: browse all work under orchestra and locate a task quickly.
|
||||||
|
|
||||||
|
### groups
|
||||||
|
|
||||||
|
- needs attention
|
||||||
|
- running
|
||||||
|
- waiting/queued
|
||||||
|
- completed
|
||||||
|
- failed
|
||||||
|
|
||||||
|
### row fields
|
||||||
|
|
||||||
|
- task title/id
|
||||||
|
- project
|
||||||
|
- current phase
|
||||||
|
- compact routing-fork phase path
|
||||||
|
- worker/harness when assigned
|
||||||
|
- last update/progress
|
||||||
|
- context occupancy for active sessions
|
||||||
|
- status
|
||||||
|
|
||||||
|
### filters
|
||||||
|
|
||||||
|
- project
|
||||||
|
- phase
|
||||||
|
- worker/harness
|
||||||
|
- status
|
||||||
|
- attention state
|
||||||
|
|
||||||
|
avoid turning this into a kanban board. orchestra's lifecycle is stateful and event-backed; the list should expose state, not encourage arbitrary drag-and-drop mutation.
|
||||||
|
|
||||||
|
## 7. task detail
|
||||||
|
|
||||||
|
mockup: `final/02-task-detail.png`
|
||||||
|
|
||||||
|
purpose: provide one calm page answering "what are we doing?", "what is happening?", and "where are we in the workflow?"
|
||||||
|
|
||||||
|
### top area
|
||||||
|
|
||||||
|
- title
|
||||||
|
- task id
|
||||||
|
- project
|
||||||
|
- current phase
|
||||||
|
- steer/correct primary action
|
||||||
|
- secondary actions menu
|
||||||
|
|
||||||
|
### what we're doing
|
||||||
|
|
||||||
|
show only:
|
||||||
|
|
||||||
|
- goal
|
||||||
|
- acceptance summary
|
||||||
|
- latest effective human decisions when present
|
||||||
|
- active constraints when present
|
||||||
|
|
||||||
|
full contract, all decisions, and historical versions open in drill-down views.
|
||||||
|
|
||||||
|
### live execution
|
||||||
|
|
||||||
|
show only:
|
||||||
|
|
||||||
|
- worker/harness
|
||||||
|
- pane
|
||||||
|
- lease remaining
|
||||||
|
- context occupancy
|
||||||
|
- last progress
|
||||||
|
- one-line current activity
|
||||||
|
|
||||||
|
links:
|
||||||
|
|
||||||
|
- open live pane
|
||||||
|
- worker logs
|
||||||
|
- launch context
|
||||||
|
- git diff
|
||||||
|
|
||||||
|
### workflow
|
||||||
|
|
||||||
|
render the ace-fca path visibly:
|
||||||
|
|
||||||
|
`frame → research → plan → implement → review → pr → merged`
|
||||||
|
|
||||||
|
completed phases are quiet green, current phase uses the orchestra accent, future phases remain neutral.
|
||||||
|
|
||||||
|
phase detail, artifacts, refusal history, and raw events belong in drill-downs rather than the default task screen.
|
||||||
|
|
||||||
|
## 8. live pane / terminal
|
||||||
|
|
||||||
|
mockup: `final/03-terminal-live-pane.png`
|
||||||
|
|
||||||
|
purpose: show the actual tmux/herdr pane, not a reconstructed log view.
|
||||||
|
|
||||||
|
### behavior
|
||||||
|
|
||||||
|
- read-only by default
|
||||||
|
- exact pane dimensions and terminal rendering preserved when possible
|
||||||
|
- fit mode may scale visually but must not alter the underlying pane
|
||||||
|
- scrollback inspection must not disturb the live cursor
|
||||||
|
- `take control` explicitly enables keyboard forwarding
|
||||||
|
- strong visual state when control is captured
|
||||||
|
- escape or a fixed shortcut releases control
|
||||||
|
|
||||||
|
### why this screen exists
|
||||||
|
|
||||||
|
it must reveal real tui state such as:
|
||||||
|
|
||||||
|
- input still sitting in the editor
|
||||||
|
- queued input
|
||||||
|
- paste placeholders
|
||||||
|
- permission/approval dialogs
|
||||||
|
- `/clear`
|
||||||
|
- `@HANDOFF.md`
|
||||||
|
- agent busy/idle presentation
|
||||||
|
- tmux status line and pane identity
|
||||||
|
|
||||||
|
### surrounding chrome
|
||||||
|
|
||||||
|
keep it minimal:
|
||||||
|
|
||||||
|
- worker/pane identity
|
||||||
|
- read-only/control state
|
||||||
|
- fit/copy/scrollback controls
|
||||||
|
- lease epoch/timing
|
||||||
|
- context occupancy
|
||||||
|
- last verified progress
|
||||||
|
- pane/worker health
|
||||||
|
- small live event strip for orchestra-originated interactions and confirmations
|
||||||
|
|
||||||
|
when a pane is lost, preserve the last captured frame and state the exact loss time and lease consequence.
|
||||||
|
|
||||||
|
## 9. decisions list
|
||||||
|
|
||||||
|
mockup: `final/05-decisions.png`
|
||||||
|
|
||||||
|
purpose: inspect durable human authority and approvals across tasks.
|
||||||
|
|
||||||
|
this is not the human attention queue; that is review.
|
||||||
|
|
||||||
|
### list fields
|
||||||
|
|
||||||
|
- decision id
|
||||||
|
- short decision text
|
||||||
|
- task
|
||||||
|
- phase
|
||||||
|
- source/provenance
|
||||||
|
- requested/recorded time
|
||||||
|
- status: active, superseded, waiting, approved, rejected, auto
|
||||||
|
- operator/actor
|
||||||
|
|
||||||
|
### filters
|
||||||
|
|
||||||
|
- active/superseded
|
||||||
|
- kind: decision, correction, constraint, answer, scope change
|
||||||
|
- project
|
||||||
|
- task
|
||||||
|
- actor/source
|
||||||
|
|
||||||
|
### detail
|
||||||
|
|
||||||
|
opening a decision should show:
|
||||||
|
|
||||||
|
- exact value
|
||||||
|
- subject
|
||||||
|
- provenance
|
||||||
|
- supersedes/superseded-by links
|
||||||
|
- event sequence
|
||||||
|
- effective authority impact
|
||||||
|
|
||||||
|
machine provenance is mono; the human decision text remains sans.
|
||||||
|
|
||||||
|
## 10. workers
|
||||||
|
|
||||||
|
mockup: `final/06-workers.png`
|
||||||
|
|
||||||
|
purpose: answer "where can work run and is that execution machinery usable?"
|
||||||
|
|
||||||
|
### worker list fields
|
||||||
|
|
||||||
|
- worker/harness identity
|
||||||
|
- online/idle/busy/offline/degraded
|
||||||
|
- harness type
|
||||||
|
- current task when leased
|
||||||
|
- phase
|
||||||
|
- lease or heartbeat age
|
||||||
|
|
||||||
|
intentionally omit generic infrastructure metrics such as disk, memory, cpu, and uptime unless orchestra directly uses them for eligibility.
|
||||||
|
|
||||||
|
### worker detail
|
||||||
|
|
||||||
|
- current session/task
|
||||||
|
- pane id
|
||||||
|
- lease epoch/remaining
|
||||||
|
- last verified progress
|
||||||
|
- worker revision
|
||||||
|
- herdr status
|
||||||
|
- last heartbeat
|
||||||
|
- projects declared by the worker
|
||||||
|
- declared capabilities if orchestra actually consumes them
|
||||||
|
|
||||||
|
### capabilities
|
||||||
|
|
||||||
|
capabilities must come from worker registration/project configuration, never be guessed from host inspection.
|
||||||
|
|
||||||
|
the ui labels them **declared capabilities** and can expose registration provenance.
|
||||||
|
|
||||||
|
### registration
|
||||||
|
|
||||||
|
registration is worker-driven:
|
||||||
|
|
||||||
|
1. worker starts with coordinator credentials/configuration
|
||||||
|
2. worker announces identity, revision, harnesses, projects, capacity and declared capabilities
|
||||||
|
3. coordinator records it and heartbeat updates liveness
|
||||||
|
|
||||||
|
`add worker` should therefore be a registration guide/bootstrap flow, not a magical browser-side creation of a remote process.
|
||||||
|
|
||||||
|
## 11. projects
|
||||||
|
|
||||||
|
mockup: `final/07-projects.png`
|
||||||
|
|
||||||
|
purpose: group long-running repository/workflow configuration and its active work.
|
||||||
|
|
||||||
|
### project list
|
||||||
|
|
||||||
|
- project name
|
||||||
|
- short description
|
||||||
|
- state
|
||||||
|
- active/total task count
|
||||||
|
- attention/decision count
|
||||||
|
- last meaningful activity
|
||||||
|
|
||||||
|
avoid aggregate artifact counts or infrastructure metrics unless they help choose an action.
|
||||||
|
|
||||||
|
### project detail
|
||||||
|
|
||||||
|
- description
|
||||||
|
- source/repository identity
|
||||||
|
- configured worker/harness affinity
|
||||||
|
- workflow phases
|
||||||
|
- trajectory-gate policy
|
||||||
|
- quality gate
|
||||||
|
- human reconcile source
|
||||||
|
- safe-operation policy
|
||||||
|
- active tasks
|
||||||
|
- recent meaningful events
|
||||||
|
|
||||||
|
configuration edits belong in the project detail, while global defaults remain in settings.
|
||||||
|
|
||||||
|
## 12. review
|
||||||
|
|
||||||
|
mockup: `final/08-review.png`
|
||||||
|
|
||||||
|
purpose: the operator inbox: **what needs me to look at something and judge?**
|
||||||
|
|
||||||
|
workers show machinery. review shows human attention.
|
||||||
|
|
||||||
|
### review item types
|
||||||
|
|
||||||
|
- trajectory decision
|
||||||
|
- human blocker / decision request
|
||||||
|
- code/pr review
|
||||||
|
- ai review requiring operator override
|
||||||
|
- operator-required lifecycle state
|
||||||
|
- closed pr without merge
|
||||||
|
- information/fyi, optionally hidden by default
|
||||||
|
|
||||||
|
### list fields
|
||||||
|
|
||||||
|
- type
|
||||||
|
- task/item
|
||||||
|
- why it is here
|
||||||
|
- project
|
||||||
|
- priority
|
||||||
|
- age
|
||||||
|
|
||||||
|
worker/pane is secondary metadata, not a primary column.
|
||||||
|
|
||||||
|
### detail drawer/page
|
||||||
|
|
||||||
|
show the minimum evidence needed to act:
|
||||||
|
|
||||||
|
- why this needs the operator
|
||||||
|
- effective task goal/constraints relevant to the decision
|
||||||
|
- proposed options when applicable
|
||||||
|
- relevant plan/diff/review evidence
|
||||||
|
- provenance and age
|
||||||
|
- comments/context drill-down
|
||||||
|
|
||||||
|
primary action names must match the lifecycle action: make decision, answer blocker, review pr, request changes, retry, etc.
|
||||||
|
|
||||||
|
snooze/defer may exist for attention management but must not silently mutate underlying task truth.
|
||||||
|
|
||||||
|
## 13. settings
|
||||||
|
|
||||||
|
mockup: `final/09-settings.png`
|
||||||
|
|
||||||
|
purpose: configure orchestra itself, project-independent defaults, integrations, policies, and operator/security settings.
|
||||||
|
|
||||||
|
### tabs
|
||||||
|
|
||||||
|
1. general
|
||||||
|
2. appearance
|
||||||
|
3. notifications
|
||||||
|
4. access & security
|
||||||
|
5. integrations
|
||||||
|
6. agents & tools
|
||||||
|
7. policies
|
||||||
|
8. advanced
|
||||||
|
|
||||||
|
### general
|
||||||
|
|
||||||
|
- instance name/environment
|
||||||
|
- locale/time display
|
||||||
|
- default task preferences
|
||||||
|
- default workflow policy
|
||||||
|
- event/artifact retention
|
||||||
|
- storage/backup state only if orchestra owns it
|
||||||
|
|
||||||
|
### appearance
|
||||||
|
|
||||||
|
- theme: system/dark/light
|
||||||
|
- ethos remains fixed; no arbitrary shell redesign controls
|
||||||
|
- accessibility and reduced motion
|
||||||
|
|
||||||
|
### access & security
|
||||||
|
|
||||||
|
- surface tokens/status
|
||||||
|
- trusted review actors
|
||||||
|
- ignored actors
|
||||||
|
- operator sessions
|
||||||
|
- agent-surface auth
|
||||||
|
- secret/config source status without exposing secret values
|
||||||
|
|
||||||
|
### integrations
|
||||||
|
|
||||||
|
- gitea
|
||||||
|
- vikunja
|
||||||
|
- notification sinks
|
||||||
|
- source reconciliation state
|
||||||
|
|
||||||
|
show endpoint/identity and last successful interaction in mono.
|
||||||
|
|
||||||
|
### agents & tools
|
||||||
|
|
||||||
|
- harness definitions
|
||||||
|
- adapter/runtime configuration
|
||||||
|
- tmux/herdr ownership
|
||||||
|
- default launch behavior
|
||||||
|
- context occupancy thresholds
|
||||||
|
|
||||||
|
### policies
|
||||||
|
|
||||||
|
- work phase defaults
|
||||||
|
- trajectory gate
|
||||||
|
- bounded decision request budget
|
||||||
|
- retry policy
|
||||||
|
- reconcile failure handoff threshold
|
||||||
|
- quality/review/submission policy
|
||||||
|
|
||||||
|
### advanced
|
||||||
|
|
||||||
|
- event-store diagnostics
|
||||||
|
- snapshots/replay
|
||||||
|
- raw configuration
|
||||||
|
- migration/status information
|
||||||
|
- destructive operator actions in a visually separated danger zone
|
||||||
|
|
||||||
|
## 14. global command surface
|
||||||
|
|
||||||
|
`cmd+k` is the shared ethos command surface and should handle navigation plus safe operator actions.
|
||||||
|
|
||||||
|
examples:
|
||||||
|
|
||||||
|
- open task by id/title
|
||||||
|
- open worker/pane
|
||||||
|
- show decisions for task
|
||||||
|
- steer/correct active task
|
||||||
|
- answer current blocker
|
||||||
|
- retry failed task
|
||||||
|
- open review item
|
||||||
|
|
||||||
|
high-risk lifecycle actions require explicit confirmation and should not be the first fuzzy-search result for ordinary text.
|
||||||
|
|
||||||
|
## 15. status grammar
|
||||||
|
|
||||||
|
status is compact and consistent across screens.
|
||||||
|
|
||||||
|
recommended semantic colors:
|
||||||
|
|
||||||
|
- green: verified healthy/completed/accepted
|
||||||
|
- violet: current/selected/orchestra-controlled active state
|
||||||
|
- amber: waiting/degraded/operator attention
|
||||||
|
- red: failed/refused/danger
|
||||||
|
- gray: idle/unknown/historical
|
||||||
|
|
||||||
|
color never carries the entire meaning; every state also has text/iconography.
|
||||||
|
|
||||||
|
## 16. evidence drill-down
|
||||||
|
|
||||||
|
any important derived claim should be traceable without ssh.
|
||||||
|
|
||||||
|
from relevant detail screens the operator should be able to reach:
|
||||||
|
|
||||||
|
- event(s) that established the state
|
||||||
|
- exact generated launch context
|
||||||
|
- sealed research/plan/review/submission artifact refs
|
||||||
|
- git sha/ref
|
||||||
|
- worker journal observations
|
||||||
|
- input confirmation receipt
|
||||||
|
- pane capture/terminal
|
||||||
|
- source comment/review provenance
|
||||||
|
|
||||||
|
raw evidence is accessible but not visible by default.
|
||||||
|
|
||||||
|
## 17. copy rules
|
||||||
|
|
||||||
|
- labels describe what the operator controls, not implementation class names
|
||||||
|
- machine values remain exact and honest
|
||||||
|
- errors state what failed and what consequence follows
|
||||||
|
- no "something went wrong"
|
||||||
|
- no spinner where a queue depth, heartbeat age, poll timestamp, retry time, or progress value exists
|
||||||
|
|
||||||
|
examples:
|
||||||
|
|
||||||
|
- `source reconcile failed · retry at 14:06:00`
|
||||||
|
- `worker heartbeat 42s old · ineligible after 60s`
|
||||||
|
- `phase request refused · next phase is research`
|
||||||
|
- `pane lost at 13:42:18 · lease expires in 09:14`
|
||||||
|
|
||||||
|
## 18. responsive behavior
|
||||||
|
|
||||||
|
### tablet/mobile priorities
|
||||||
|
|
||||||
|
keep visible:
|
||||||
|
|
||||||
|
1. attention state
|
||||||
|
2. task title/phase
|
||||||
|
3. latest human authority
|
||||||
|
4. live execution state
|
||||||
|
5. primary action
|
||||||
|
|
||||||
|
collapse or drill down:
|
||||||
|
|
||||||
|
- long acceptance lists
|
||||||
|
- complete workflow history
|
||||||
|
- detailed lease metadata
|
||||||
|
- worker journal
|
||||||
|
- raw events
|
||||||
|
- project configuration
|
||||||
|
|
||||||
|
terminal becomes its own full-screen route and should not be squeezed into a card.
|
||||||
|
|
||||||
|
## 19. implementation order
|
||||||
|
|
||||||
|
recommended web-ui build order:
|
||||||
|
|
||||||
|
1. ethos shell + orchestra tokens/motif
|
||||||
|
2. dashboard
|
||||||
|
3. tasks list
|
||||||
|
4. task detail
|
||||||
|
5. live pane
|
||||||
|
6. review inbox
|
||||||
|
7. decisions list/detail
|
||||||
|
8. workers
|
||||||
|
9. projects
|
||||||
|
10. settings
|
||||||
|
11. global command surface
|
||||||
|
12. evidence/raw-event drill-down
|
||||||
|
|
||||||
|
build each screen against real orchestra api data rather than static ui-only state as early as practical.
|
||||||
|
|
||||||
|
## 20. verification
|
||||||
|
|
||||||
|
for every screen:
|
||||||
|
|
||||||
|
- render and visually inspect pixels
|
||||||
|
- check desktop and <=640px mobile layout
|
||||||
|
- assert no horizontal overflow
|
||||||
|
- verify mono/sans split visibly
|
||||||
|
- verify accent is signal-only
|
||||||
|
- verify no blur/backdrop-filter exists
|
||||||
|
- verify empty/error/loading states show real machine state
|
||||||
|
- compare implemented screen against the corresponding mockup as direction, not pixel-perfect contract
|
||||||
|
|
||||||
|
mockups communicate hierarchy, density, and interaction intent. orchestra state and the ethos design laws are the actual specification.
|
||||||