Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| de18f372d3 | |||
| bdc0d4d5be | |||
| 6cb2f932d8 | |||
| b5c37f693b | |||
| bbf3335857 | |||
| 99b209ba10 | |||
| ab3258833d | |||
| c587f2cc8d | |||
| 41658aea5b | |||
| 765bf2afc6 | |||
| cbd6b11c49 |
@@ -2527,3 +2527,324 @@ transaction and no session behind, on either worker.
|
||||
|
||||
Both workers hold no sessions and no release transactions. Deployed pair is
|
||||
still `3c7cf95` on both halves.
|
||||
|
||||
## Run 16, 2026-08-29: the rotation rung, half proven
|
||||
|
||||
Task `06G4SWEVP71FYYKAV5FV0ZK5ZG` on `34f3c28`, a three-phase plan
|
||||
(`78d25730140c`) with automated checks inside policy.
|
||||
|
||||
### Proven: a successor inherits the whole sealed plan
|
||||
|
||||
The launch context of the session that picked the task up after its rotation
|
||||
carries the complete plan: overview, current state, desired end state,
|
||||
non-goals, approach, all three phases with their files, changes and
|
||||
verification, testing strategy, risks, migration, and ten research citations.
|
||||
|
||||
### Not a defect: that successor was not given phase progress
|
||||
|
||||
`renderPlanProgress` runs only for `WorkPhaseImplement`
|
||||
(`internal/agentctx/agentctx.go:321`). This successor picked up in review, and
|
||||
the code says why: an independent review must reconstruct the change from the
|
||||
diff rather than inherit the implementer's account. Progress is withheld there
|
||||
deliberately.
|
||||
|
||||
So the rung still needs an **implement-phase** successor. The records exist and
|
||||
are durable, projected by `Task.PlanPhases()`.
|
||||
|
||||
### Why the mid-implement rotation missed
|
||||
|
||||
A trivial task spends about four minutes in implement, and all three phases
|
||||
verified inside two and a half of them, every one against the same tree
|
||||
`3da86dbf8ad5`. The implementer does the whole change first, then verifies each
|
||||
phase. The suspend, edit and restart that sets `handoff_requested` takes about
|
||||
a minute, and it landed after the implementer had already asked for review.
|
||||
|
||||
Two mechanics worth keeping:
|
||||
|
||||
- **The web handoff action cannot drive this.** `RequestHandoff` needs a local
|
||||
coordinator and every session here is worker-owned, so it answers 503. That
|
||||
is the Design A guardrail working, and a fourth UI-exposed gap.
|
||||
- **A state-file edit does not survive a running worker.** It holds sessions in
|
||||
memory and writes them back. Suspend, edit, then restart; resuming lets the
|
||||
old copy win. This cost one attempt to learn.
|
||||
|
||||
### F62: a requested handoff that is never answered is invisible
|
||||
|
||||
The rotation is agent-driven: the worker asks, and the agent must write
|
||||
`HANDOFF.md` at a turn boundary. The review agent never did. Renewals stopped,
|
||||
the lease expired at 10:12:05, and the task requeued having lost an attempt.
|
||||
|
||||
Worker health recorded exactly one thing, 34 times:
|
||||
|
||||
```text
|
||||
x34 lease 06G4SWEVP71FYYKAV5FV0ZK5ZG not renewed: agent status idle and pane unchanged
|
||||
```
|
||||
|
||||
Nothing says a handoff was requested and left unanswered. There is no timeout,
|
||||
no retry, and no observation. The only visible consequence is an expiry that
|
||||
looks identical to an ordinary idle one. Diagnosing it needed the worker state
|
||||
file.
|
||||
|
||||
The ring earned its keep again: one distinct message with a count of 34, rather
|
||||
than 34 overwrites of one slot.
|
||||
|
||||
**Fixed, not yet proven live.** The request is now stamped
|
||||
(`herdr.Session.HandoffRequestedAt`) and the wait is bounded by
|
||||
`watchHandoff` in the worker:
|
||||
|
||||
- the lease renews while Orchestra is explicitly waiting, because a quiet pane
|
||||
is the answer the agent was asked for — the renewal gate's new case, bounded
|
||||
by `handoffAnswerTimeout`;
|
||||
- the request is re-sent once at `handoffRetryAfter` (4 minutes), with the same
|
||||
reason it was first asked with;
|
||||
- at 10 minutes the worker nacks with failure class `handoff_unanswered`, and
|
||||
the coordinator emits `TaskReleased reason=handoff_unanswered` rather than
|
||||
letting the lease die as generic idleness.
|
||||
|
||||
`DebtClassForFailureClass` knows the class, so a harness that repeatedly
|
||||
ignores handoff requests now accumulates in the debt ledger instead of hiding
|
||||
inside `lease_expired`.
|
||||
|
||||
Runtime proof still owed: force a request the agent will not answer, and read
|
||||
the release event rather than the worker state file.
|
||||
|
||||
## Run 17, 2026-08-29: F62 proven live on `c587f2c`
|
||||
|
||||
Task `06G4V20T528ZTER7KZBVGNAYXC`, a five-phase plan
|
||||
(`3b16b77054b9`) on the deployed pair, coordinator and worker both at
|
||||
`c587f2c`.
|
||||
|
||||
### The trigger was production, not a rig
|
||||
|
||||
The rig planned for this run was a state-file lever. It was never needed. The
|
||||
`implement -> review` phase change asked for a handoff through the ordinary
|
||||
path (`rotateForPhase`, reason `phase_changed`), and the agent ignored it
|
||||
because this task's brief instructed it to from phase 4 onward. Nothing touched
|
||||
tmux, no process was stopped, and the pane stayed open for the whole wait.
|
||||
|
||||
```text
|
||||
12:48:09 implement -> review, handoff requested, reason=phase_changed
|
||||
12:50:09 TaskLeaseRenewed agent idle, pane capture unchanged
|
||||
12:52:15 pane input confirmed the single resend, 4m06s after the request
|
||||
12:52:39 TaskLeaseRenewed
|
||||
12:55:14 TaskLeaseRenewed
|
||||
12:57:49 TaskLeaseRenewed
|
||||
12:58:14 TaskReleased reason=handoff_unanswered failure_class=handoff_unanswered
|
||||
last_error="handoff requested (phase_changed) and unanswered for 10m5s"
|
||||
12:59:14 TaskLeased successor leases normally
|
||||
13:02:20 TaskSubmitted
|
||||
```
|
||||
|
||||
Every assertion the operator set for this rung held:
|
||||
|
||||
- the release does not look like an ordinary `lease_expired`: it carries
|
||||
`reason=handoff_unanswered` and `lifecycle_phase=handoff_unanswered`;
|
||||
- the idle-renewal rule did not kill the wait: four renewals on an idle pane
|
||||
with no capture movement, each one the gate would have refused before this
|
||||
commit;
|
||||
- the resend preserved the reason the request was first made with;
|
||||
- exactly one resend, not one per tick;
|
||||
- the timeout produced the class at `10m5s`, one tick past the bound;
|
||||
- nothing stale was left: `releases: {}`, no session, no lease, and zero tmux
|
||||
panes matching the id;
|
||||
- the next attempt leased 60s later and reached `TaskSubmitted`.
|
||||
|
||||
The debt projection classified it with no new code beyond the class mapping:
|
||||
|
||||
```text
|
||||
v1:operational:handoff_unanswered:workpc-claude:lease operational important
|
||||
"handoff requested (phase_changed) and unanswered for 10m5s"
|
||||
```
|
||||
|
||||
**Inference, not observation.** The resent prompt's text was not captured. That
|
||||
it carried the `phase_changed` wording rests on the code path and the timing,
|
||||
not on a pane capture. A future rig should publish a capture across the resend.
|
||||
|
||||
### The implement-successor rung missed again, and why
|
||||
|
||||
A five-phase plan does not lengthen the implement phase. This one ran 12:45:18
|
||||
to 12:48:09, about 2m50s, with verifications 20 to 40 seconds apart:
|
||||
|
||||
```text
|
||||
12:45:47 phase-1 12:46:07 phase-2 12:46:32 phase-3
|
||||
12:47:12 phase-4 12:47:52 phase-5 12:48:09 implement -> review
|
||||
```
|
||||
|
||||
The brief's explicit one-phase-at-a-time requirement was followed, and it
|
||||
still took under three minutes. Phase count is not the lever; volume of
|
||||
mechanical work is.
|
||||
|
||||
`awaiting_manual_verification` does not hold the phase open either. All five
|
||||
phases carried that status and the agent moved on regardless.
|
||||
|
||||
### The state-file lever cannot be driven with `restart` alone
|
||||
|
||||
The documented procedure is stop, edit, start. The unattended sudo rule grants
|
||||
`install` and `systemctl restart orchestra-worker`, with no `stop` or `start`.
|
||||
With `restart` only, the running worker's five-second tick save landed between
|
||||
the edit and the restart, and the edit was gone. Widening the sudoers line is
|
||||
the fix.
|
||||
|
||||
### One cosmetic note
|
||||
|
||||
`Session.HandoffRequestedAt` serializes as `0001-01-01T00:00:00Z` when unset,
|
||||
because `omitempty` does not omit a zero `time.Time`. The worker parses it back
|
||||
to a zero value and `watchHandoff` starts the clock, so behaviour is correct.
|
||||
It is the same marshalling trap the operator console hit.
|
||||
|
||||
## Run 18, 2026-08-29: the implement-successor rung, proven
|
||||
|
||||
Task `06G4VF5HZW7Q4JBM3TTY7W1Y64` on `c587f2c`, a five-phase plan whose phases
|
||||
are ten named checks rather than one refactor. Volume of mechanical work is
|
||||
what keeps the implement phase open; phase count does not.
|
||||
|
||||
### The rotation
|
||||
|
||||
```text
|
||||
13:43:10 implement session launched
|
||||
13:43:44 phase-1 verified
|
||||
13:44:04 phase-2 verified
|
||||
13:44:35 handoff requested, reason=milestone, delivered by watchHandoff's resend
|
||||
13:45:25 TaskLeaseRenewed
|
||||
13:46:51 TaskReleased handoff_ref 0b4fae6f705a anchor 8ea3c7d1ae8f
|
||||
13:46:51 TaskLeased successor, same harness
|
||||
13:46:55 TaskPickupValidated
|
||||
13:47:06 TaskLaunchAcknowledged, still in implement
|
||||
```
|
||||
|
||||
The lever was the worker state file, backdated past `handoffRetryAfter` so the
|
||||
request went out on the next tick. Under `c587f2c` that lever now produces a
|
||||
real prompt: the flag alone used to sit there unasked, which is how run 16
|
||||
ended in a silent expiry.
|
||||
|
||||
### What the successor was given
|
||||
|
||||
From `.orchestra/launch.md`, written at 13:47:05:
|
||||
|
||||
```text
|
||||
## Current phase
|
||||
implement: Implement the accepted plan below. Verify as you go.
|
||||
|
||||
## Verified git state
|
||||
- head: 8ea3c7d1ae8fd960897344694645117adbd8ee82
|
||||
- uncommitted changes: false
|
||||
|
||||
## Current human decisions
|
||||
None recorded. Work from the goal and acceptance above.
|
||||
|
||||
## Plan progress
|
||||
Orchestra established this by running the plan's own verification. You cannot
|
||||
write it.
|
||||
- phase-1 (...): automated checks passed at d3acd4989a20, stale because the
|
||||
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
|
||||
- phase-2 (...): automated checks passed at d3acd4989a20, stale because the
|
||||
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
|
||||
- phase-3 (Checks 7 to 8): not started
|
||||
- phase-4 (...): not started
|
||||
- phase-5 (...): not started
|
||||
```
|
||||
|
||||
Everything the rung asked for is there: the complete accepted plan with all
|
||||
five phases, their files, changes and verification; the phases already
|
||||
verified; the first unfinished phase; and the current human authority. The
|
||||
launch context also carries the accepted research and its dead ends.
|
||||
|
||||
**SHA staleness renders itself.** Neither phase is reported as simply passed.
|
||||
Each says the checks passed at `d3acd4989a20` and are stale because the tree
|
||||
has moved to `8ea3c7d1ae8f`. That is half of the manual-verification rung
|
||||
observed without being asked for.
|
||||
|
||||
### The lever still loses a race, sometimes
|
||||
|
||||
`systemctl restart` alone leaves a five-second window in which the running
|
||||
worker's tick save can clobber the edit. It clobbered run 17's attempt and
|
||||
survived run 18's. Stop, edit, start is the reliable sequence, and it needs
|
||||
`systemctl stop` and `start` in the unattended sudo rule.
|
||||
|
||||
## Run 19, 2026-08-29: the manual verification seam, and F63
|
||||
|
||||
Two tasks, both ingested from Gitea so that comments reconcile:
|
||||
`06G4W2TMZ9MW9SEBM5MGTM5288` (issue 24, plan `500464b9`) and
|
||||
`06G4W63545T3RV8SSKGKN11B3G` (issue 26, plan `53f7918b`).
|
||||
|
||||
A burn-in task cannot test this rung. Comments reconcile only for the source
|
||||
the task came from, so a task created with `source: "burnin"` has no comment
|
||||
stream at all.
|
||||
|
||||
### Proven: the keyed seam
|
||||
|
||||
```text
|
||||
15:09:15 PlanPhaseVerified phase-1 awaiting_manual_verification at 80728967
|
||||
comment: "Looks good to me. Nice work on this one, ship it."
|
||||
17:10:03 HumanDecisionRecorded subject=operator_instruction
|
||||
phase-1 unchanged, still awaiting_manual_verification
|
||||
comment: "orchestra verify phase-1"
|
||||
17:10:25 HumanDecisionRecorded subject=plan_phase_verification:500464b9…:phase-1
|
||||
phase-1 verified
|
||||
```
|
||||
|
||||
All five phases reached `awaiting_manual_verification` on their automated
|
||||
checks alone. A generic approval satisfied nothing. The keyed form flipped
|
||||
exactly the phase it named, under the plan ref Orchestra supplied rather than
|
||||
one the commenter chose.
|
||||
|
||||
### Proven: re-verification at a new SHA
|
||||
|
||||
On issue 26, inside one live implement lease, with the worker stopped for the
|
||||
whole edit:
|
||||
|
||||
```text
|
||||
15:23:00 PlanPhaseVerified phase-1 awaiting_manual_verification at f5d8f90b
|
||||
operator commit moves the tree f5d8f90b -> 1f07b89c
|
||||
.orchestra/plan-progress.json requests phase-1 again
|
||||
15:23:17 PlanPhaseVerified phase-1 awaiting_manual_verification at 1f07b89c
|
||||
17:23:16 HumanDecisionRecorded subject=plan_phase_verification:53f7918b…:phase-1
|
||||
phase-1 verified at 1f07b89c
|
||||
```
|
||||
|
||||
The rerun ran the plan's own commands at the new tree and wrote a fresh record
|
||||
bound to it. The worker never reads a command out of the request: the request
|
||||
names a phase and nothing else about it is trusted.
|
||||
|
||||
### F63: a manual sign-off outlives the tree it was given against
|
||||
|
||||
The second half of the same rig, with no new comment:
|
||||
|
||||
```text
|
||||
operator commit moves the tree 1f07b89c -> 28af84ff
|
||||
.orchestra/plan-progress.json requests phase-1 again
|
||||
15:24:02 PlanPhaseVerified phase-1 verified at 28af84ff
|
||||
```
|
||||
|
||||
`RecordPlanPhaseVerification` consults `manuallySignedOff`
|
||||
(`internal/operations/planprogress.go:127,152`), which asks only whether a
|
||||
decision with that subject exists. It does, forever, for that plan ref and
|
||||
phase id. So the automated half of a phase is re-established at every new SHA
|
||||
while the manual half is asserted from a tree the human never saw.
|
||||
|
||||
This is the same class of bug `AtSHA` exists to prevent, applied to the half
|
||||
that `AtSHA` does not cover. A manual check on this project is a human reading
|
||||
printed output; an edit between the sign-off and the rerun can change exactly
|
||||
that output.
|
||||
|
||||
Only a replan clears it, because `PlanPhases()` drops records whose plan ref is
|
||||
not the accepted one.
|
||||
|
||||
Proposed shape, not built: when the reducer flips a record on a keyed decision
|
||||
(`internal/store/store.go:273`), it knows the record it satisfied and that
|
||||
record's `AtSHA`. Stamp that sha on the decision, and have `manuallySignedOff`
|
||||
require it to match the run being recorded. A sign-off that arrives before any
|
||||
run has no sha and keeps today's behaviour, so the ordinary ordering is
|
||||
unchanged.
|
||||
|
||||
### Also established
|
||||
|
||||
- `review -> implement` is legal in `legalPhaseTransitions`, but
|
||||
`AdvanceWorkPhase` only walks the project's path forward. A task that has
|
||||
left implement can return only through a review that requests changes, not
|
||||
through the operator phase endpoint.
|
||||
- The agent's re-verification request is a file an operator can write:
|
||||
`.orchestra/plan-progress.json`, `{"phase": "...", "status":
|
||||
"ready_for_verification"}`. It is the whole lever for this rung.
|
||||
- Ingest turns every issue label into a required capability, so a rig issue
|
||||
must carry no labels or it will not schedule.
|
||||
|
||||
@@ -66,6 +66,14 @@ the live herdr instance and check.
|
||||
`docker logs orchestra-api`. **Deploying a code change means rebuilding the
|
||||
compose images** (`up -d --build`) — the running image can silently predate
|
||||
recent commits, so compare its build time against `git log`.
|
||||
- **A coordinator deploy does not deploy the console.** They are two images
|
||||
built from the same repo, and the usual `up -d --no-deps orchestra-api`
|
||||
leaves `orchestra-web-ui` on whatever it was. This has bitten twice: the
|
||||
ethos console landed in the repo on 2026-08-29 02:44 and was still serving a
|
||||
2026-07-30 image hours later, through two coordinator deploys. Rebuild it
|
||||
explicitly from the same clean worktree (`docker build` in `web/`, then
|
||||
`up -d --no-deps orchestra-web-ui`), and check the served bundle rather than
|
||||
the commit: `curl -s http://127.0.0.1:19145/assets/<css> | grep 8F7AE5`.
|
||||
- `orchestra.service` was the **previous** deployment; the unit file was
|
||||
deleted from `deploy/` on 2026-07-31 along with `redeploy.sh` (which
|
||||
`sudo install`ed to `/usr/local/bin` and restarted it). If a stale copy is
|
||||
|
||||
@@ -0,0 +1,349 @@
|
||||
# Handoff: the release path is settled, and the UI became a truth detector
|
||||
|
||||
Written 2026-08-29, 14:20 local (10:20 UTC). Read with `BURNIN.md` (the run
|
||||
ledger, current through run 16), `DEBT-DESIGN.md`, `PLAN-SPEC-DESIGN.md`,
|
||||
`AUDIT.md` and `CLAUDE.md`. The previous handoff is
|
||||
`HANDOFF-2026-08-28-plan-v1.md`.
|
||||
|
||||
Everything below was observed live unless it says otherwise.
|
||||
|
||||
## The headline
|
||||
|
||||
**The expired-release defect is fixed and proven both ways.** The whole family
|
||||
around it is closed. F57 through F60 settle what happens to a release
|
||||
transaction in every case. That includes the ones that used to need an operator
|
||||
with a text editor.
|
||||
|
||||
**Three new things exist that did not before.** A bounded observation ring on
|
||||
worker health, closing F18. A read-only debt ledger projected from the event
|
||||
log. An operator console rebuilt on the ethos design system.
|
||||
|
||||
**The UI turned out to be a truth detector.** Nine screens were built against
|
||||
real endpoints. They found four places where Orchestra has no capability to
|
||||
support the intended interface. That list is the most valuable output of the
|
||||
session.
|
||||
|
||||
```text
|
||||
orchestra-plan-v1 plan machinery proven
|
||||
↓
|
||||
orchestra-f18-baseline d6ee10f, the bounded observation ring
|
||||
↓ 6 commits
|
||||
34f3c28 deployed now: release path settled, debt ledger, new UI
|
||||
```
|
||||
|
||||
## Deployed state
|
||||
|
||||
| Half | Revision |
|
||||
|---|---|
|
||||
| Coordinator, homesrv container | `34f3c28` |
|
||||
| Worker, workpc systemd | `34f3c28` |
|
||||
|
||||
```text
|
||||
commit 34f3c2888fc7d45d190d93e1ea42501b6cd3e474
|
||||
coordinator sha256 1d32d83ec859b36e12473fab01bbfc3c769b97b4f6ae4db2841db5161e0eda19
|
||||
worker sha256 0e3877321dea8a1eeda51ccc6f3ead1a14aa5f5cae4d95b704f61c364248ec65
|
||||
```
|
||||
|
||||
`cbd6b11` is one commit above and is documentation only. Three commits are
|
||||
unpushed.
|
||||
|
||||
**Worker installs no longer need a human.** The operator installed the
|
||||
`/etc/sudoers.d` line, so `sudo -n install …` and
|
||||
`sudo -n systemctl restart orchestra-worker` both work unattended. Verify the
|
||||
running revision from the journal, never the installed file.
|
||||
|
||||
## The defects fixed, and how each was found
|
||||
|
||||
Not one came from reading code. Every one came from a live run failing.
|
||||
|
||||
| Id | Commit | What |
|
||||
|---|---|---|
|
||||
| F57 | `6565b9f` | An expired lease could never commit the anchor it had already pushed. The worker sent an epoch the expiry replay had deleted, and the coordinator refused any handoff without a live lease. The epoch now belongs to the transaction, `TaskReleased` retains the ending epoch, and `lateHandoffAccepted` lets exactly that owner commit while the task is queued and unleased. |
|
||||
| F58 | `03663f4` | A superseded transaction retried a permanent 409 forever, holding the pane and pinning `ActiveTask`. Run 10's task did it for seven hours. `TaskLeased` now abandons a transaction whose id the lease does not carry. |
|
||||
| F59 | `8e37989` | F58 fires on `TaskLeased`, and a failed task is never leased again. `TaskFailed` now drops the transaction too. |
|
||||
| F60 | `3c7cf95` | The general rule the other two were reaching for. Terminal is failed or completed. Blocked keeps the transaction, because a reopen can still commit it, so `TaskBlocked` now retains the ending epoch as well. A refusal parks the commit for 30s backing off to 5 minutes, and any event about the task un-parks it. A transport failure is not an answer and retries at once. |
|
||||
| F18 | `d6ee10f` | The single `last_error` slot. Worker health now carries up to sixteen distinct observations with repeat counts and first/last times, collapsing by message rather than by position. |
|
||||
|
||||
### The rig that proved F57, and the guard
|
||||
|
||||
```text
|
||||
19:00:10.742 transaction opens at prepared, anchor pushing
|
||||
19:00:10.727 TaskReleased v15 reason=lease_expired surface=tui
|
||||
19:00:11.662 TaskReleased v16 the late commit, accepted 935ms after the lease died
|
||||
19:00:11.665 TaskLeased v17 successor picks up the handoff
|
||||
19:00:15.524 TaskPickupValidated v18
|
||||
```
|
||||
|
||||
Race guard, next boundary: force the expiry, then lease the task to a probe
|
||||
harness before the push finishes. The late commit is refused, no handoff is
|
||||
written, and the successor's lease stands.
|
||||
|
||||
**The rig technique matters more than the rig.** Suspending the worker cannot
|
||||
produce this ordering. The event replay runs at the top of every tick and
|
||||
discards the transaction. The ordering exists only inside one call:
|
||||
transaction opened, anchor pushing, commit not yet sent. So poll the worker
|
||||
state file at 2ms and fire `POST /v1/tasks/<id>/release` the instant a
|
||||
transaction appears at `prepared`.
|
||||
|
||||
**A named probe harness owns a lease without starting an agent.**
|
||||
`race-guard-probe` never picks anything up and expires on the normal TTL.
|
||||
|
||||
## Corrections to the previous handoff
|
||||
|
||||
**The operator lifecycle actions do not lose a version race.** On a leased
|
||||
task, `block`, `release` and `attention` are refused by
|
||||
`internal/store/store.go:885-901` when the payload omits `harness_id` and
|
||||
`lease_epoch`. Ten attempts in 550ms all failed that way. Send both fencing
|
||||
fields and they succeed on the first try.
|
||||
|
||||
**The OpenCode Zen free tier is not blocked.** The selected model was.
|
||||
|
||||
## The debt ledger
|
||||
|
||||
`DEBT-DESIGN.md` answers nine design questions and carries four amendments the
|
||||
operator made. Slice one is built, deployed and run.
|
||||
|
||||
**Slice one writes nothing.** A read-only projection over the existing log,
|
||||
plus a pure eligibility function and `GET /v1/debt`. It folded 881 events and
|
||||
produced 15 candidates and 3 gaps.
|
||||
|
||||
```text
|
||||
v1:operational:lease_expired:workpc-opencode:lease r=41 tasks=4
|
||||
v1:operational:lease_expired:workpc-claude:lease r=29 tasks=13
|
||||
v1:operational:lease_failure:-:lease r=20 tasks=16
|
||||
v1:correctness:handoff_validation:-:lease r=5 tasks=5
|
||||
```
|
||||
|
||||
The opencode failure shape is the top item, found mechanically. Run 14 reached
|
||||
the same conclusion by hand from a pane capture.
|
||||
|
||||
**It reported what it cannot see, which was the point.** The 409 release loop
|
||||
does not appear. That evidence lived in the F18 worker ring, and no event
|
||||
carries it. Manual interventions are a non-durable gap for the same reason.
|
||||
|
||||
**The first run exposed four defects in the model**, all recorded at the end of
|
||||
`DEBT-DESIGN.md`:
|
||||
|
||||
- the component part is too coarse for lease evidence
|
||||
- `harness` is often empty on block reasons
|
||||
- path normalization mangled a mismatch reference
|
||||
- recurrence alone is the wrong sort order
|
||||
|
||||
Layout, so the read model does not end up in the command layer:
|
||||
|
||||
```text
|
||||
internal/domain/debt.go types, signature, classification
|
||||
internal/store/debt_projection.go the fold, and the gap report
|
||||
internal/operations/debt.go CheckDebtEligibility
|
||||
```
|
||||
|
||||
## The operator console
|
||||
|
||||
Nine screens on the ethos system, signal violet `#8F7AE5`, routing fork motif.
|
||||
Each screen was built by its own agent against a foundation with one author.
|
||||
The shell, tokens and primitives could not drift into nine dialects.
|
||||
|
||||
**Render before signing off.** Three bugs existed that no computed value would
|
||||
have caught. All three came from looking at a screenshot:
|
||||
|
||||
- The previous stylesheet fought every shared class name and leaked properties
|
||||
the new rules never mention, which is how `position: fixed` survived on
|
||||
`.topbar`. It is now scoped under `.legacy` and reaches only the login route.
|
||||
That also stops its green accent and its `backdrop-filter` from reaching the
|
||||
console.
|
||||
- Go marshals a zero `time.Time` as `0001-01-01T00:00:00Z` and `omitempty` does
|
||||
not omit a struct, so absent timestamps arrived populated-looking and
|
||||
rendered as `739855d ago`. Stripped once in `client.ts`, with a test.
|
||||
- Long machine ids overflowed their cards and painted under the next one.
|
||||
|
||||
Chromium is installed at `/usr/bin/chromium`. To see a screen without a live
|
||||
session, write a throwaway harness that stubs `window.fetch` and renders
|
||||
`<Console>` inside a `MemoryRouter`, served by vite on a spare port. Note that
|
||||
`npx` and `./node_modules/.bin/*` do not work on this filesystem: call
|
||||
`node ./node_modules/vite/bin/vite.js` directly.
|
||||
|
||||
## What the UI proved Orchestra cannot do
|
||||
|
||||
This is the part worth acting on. Each screen refused to fake something, and
|
||||
the refusals name real capability gaps.
|
||||
|
||||
| Gap | Evidence |
|
||||
|---|---|
|
||||
| **No web-facing human-decision write path** | `Steer / Correct` is disabled. `internal/ui/ui.go`'s action switch has grant/deny approval, resubmit, handoff, release, block and complete, and nothing writes `HumanDecisionRecorded`. The spec makes steering the primary action of the task detail screen. |
|
||||
| **No keystroke forwarding** | `Take control` is disabled. Only resubmit and approval grant/deny reach a live pane. |
|
||||
| **Context occupancy is trapped in herdr** | Three screens independently hit it. No projection carries it. |
|
||||
| **Project configuration is not served** | Repo, remote, quality gate and verification policy live only in `config.jsonc`. The projects screen can show none of it. |
|
||||
| **The web cannot request a handoff for a federated task** | `RequestHandoff` needs a local coordinator and answers 503. That is the Design A guardrail working. |
|
||||
|
||||
The operator's direction on these. Treat first-class direct human input as the
|
||||
highest-value backend feature. Build it as `POST /v1/tasks/<id>/decisions`,
|
||||
using the same durable decision semantics as Gitea comments, so Gitea, CLI and
|
||||
web converge on one `HumanDecisionRecorded`. Keep take-control disabled,
|
||||
because arbitrary pane input bypasses the durable authority model. Expose
|
||||
occupancy through a session health projection rather than teaching the web
|
||||
server about herdr. Add a read-only effective project configuration endpoint,
|
||||
which F61 will also need.
|
||||
|
||||
## Where the plan-machinery ladder stands
|
||||
|
||||
Proven in run 14: the worker executes the **sealed plan's** commands rather
|
||||
than the request's, and every `PlanPhaseVerified` binds `plan_ref`, `phase_id`,
|
||||
`at_sha`, `evidence_ref`, `lease_epoch` and `harness_id`.
|
||||
|
||||
Proven in run 16: a successor inherits the **whole sealed plan**, all phases
|
||||
with their files, changes, verification, and the research citations.
|
||||
|
||||
**Still unproven, and the next runtime item:**
|
||||
|
||||
```text
|
||||
mid-implement rotation
|
||||
→ successor picks up in implement
|
||||
→ launch context states which phases are already verified
|
||||
|
||||
manual verification
|
||||
→ SHA goes stale
|
||||
→ re-verification
|
||||
|
||||
plan mismatch
|
||||
→ human decision
|
||||
→ real replan, old plan retained, replacement launched
|
||||
```
|
||||
|
||||
Two things make the first one hard, and both are now known:
|
||||
|
||||
- **A trivial task spends about four minutes in implement**, and verifies every
|
||||
phase against one tree near the end. The implementer writes the whole change
|
||||
first, then verifies each phase in turn. Use a task whose implement phase
|
||||
genuinely runs long.
|
||||
- **A state-file edit does not survive a running worker.** It holds sessions in
|
||||
memory and writes them back. Suspend, edit, then `sudo systemctl restart`.
|
||||
Resuming lets the old copy win. Setting `handoff_requested` on the session is
|
||||
the production rotation lever.
|
||||
|
||||
Progress is withheld from a **review** successor on purpose
|
||||
(`internal/agentctx/agentctx.go:321`), because an independent review must
|
||||
reconstruct the change from the diff. That absence is not the defect.
|
||||
|
||||
## F61 and F62, recorded and not built
|
||||
|
||||
**F61: the planner learns the verification policy by refusal.** The brief says
|
||||
a policy exists, not what is in it. Every plan therefore pays one refused round
|
||||
trip. The recovery loop works: run 14's planner consumed the refusal and
|
||||
resealed 31 seconds later. It is not a one-liner, because `agentctx.Input.Policy` is filled
|
||||
from the worker's `SafeOperations` while the verification policy is
|
||||
coordinator-side. The reason to promote it later is local models, which may
|
||||
propose forbidden commands repeatedly because they cannot infer the allowed
|
||||
substitute.
|
||||
|
||||
**F62: a requested handoff nobody answers is invisible.** The rotation is
|
||||
agent-driven. In run 16 the agent never wrote `HANDOFF.md`, renewals stopped,
|
||||
the lease expired, and the task lost an attempt. Worker health recorded only
|
||||
`agent status idle and pane unchanged`, 34 times. There is no timeout, no
|
||||
retry, and no observation saying a handoff was requested and left unanswered.
|
||||
The expiry is indistinguishable from an ordinary idle one.
|
||||
|
||||
## Live state
|
||||
|
||||
Three pull requests are open and unreviewed:
|
||||
|
||||
```text
|
||||
06G4M6HF1Z3EREX1X3NEKSHP24 pulls/18
|
||||
06G4M8WHGQ4P3GQMPEEH0RJRHM pulls/19
|
||||
06G4SWEVP71FYYKAV5FV0ZK5ZG pulls/20
|
||||
```
|
||||
|
||||
Both workers are online on `34f3c28` with no release transactions and no
|
||||
sessions. 29 blocked `test-e2e` tasks are burn-in debris. Three `correx` tasks
|
||||
are queued and unschedulable, because `correx` has no entry in the
|
||||
coordinator's `config.jsonc`.
|
||||
|
||||
**opencode now runs.** The model was the problem, and `hy3-free` works. It then
|
||||
stops on a permission prompt, because `~/.config/opencode/opencode.jsonc` sets
|
||||
`"bash": "ask"`. Orchestra can answer that exact dialog, but only when an
|
||||
operator queues `grant_approval`, so unattended work stalls on the first
|
||||
command. Set `"bash": "allow"` for unattended runs. That file also has no
|
||||
top-level `model` key, so OpenCode picks whatever sits at the top of
|
||||
`~/.local/state/opencode/model.json`, which any manual pick silently changes.
|
||||
|
||||
**The opencode adapter is now a debt item, not a curiosity.** It cannot resolve
|
||||
a session file. Activity therefore reads `unknown`, and the worker cannot tell
|
||||
finished from never-started. The debt projection surfaced it independently as
|
||||
the top recurring operational item.
|
||||
|
||||
## Things that will bite
|
||||
|
||||
- **Background python tasks get killed here.** Three watchers died before doing
|
||||
anything. Foreground polling and the `Monitor` tool both work.
|
||||
- **`/v1/events` is one line of JSON.** A `grep` for two substrings matches
|
||||
across unrelated tasks. Parse it.
|
||||
- **`npx` and `./node_modules/.bin/*` fail on this filesystem.** Call node
|
||||
directly.
|
||||
- **`rm` and `cp` are interactive.** Use `/bin/rm -f` and `install`.
|
||||
- **Secrets are guarded.** Expand a token inside the container in one remote
|
||||
command:
|
||||
`T=$(docker exec orchestra-api printenv ORCHESTRA_TUI_TOKEN); curl -s -H "X-Orchestra-Surface: tui" -H "Authorization: Bearer $T" ...`
|
||||
- **Rebuild the coordinator from a detached worktree**, and split the worktree
|
||||
add, the docker build and the compose up into separate commands.
|
||||
- **`deploy/build.sh` builds both halves from one commit with one stamp.** Use
|
||||
it.
|
||||
- **Geist is not on disk.** Both stacks fall back to system faces, and the
|
||||
ethos threat model rules out the font CDN.
|
||||
|
||||
## Resume in this order
|
||||
|
||||
Set by the operator at the session boundary. Do not rebuild it from the commits.
|
||||
|
||||
1. **F62 first.** Make a requested-but-unanswered handoff visible and bounded.
|
||||
Preserve the lease while Orchestra is explicitly waiting, retry the confirmed
|
||||
request, then emit a causal timeout. Today it ends as generic idleness.
|
||||
2. **Repeat the implement-successor rung** with a deliberately longer task. Get
|
||||
at least one `PlanPhaseVerified`, force the handoff while still in
|
||||
`implement`, and prove the successor sees the complete accepted plan, the
|
||||
verified previous phase, the first unfinished phase, and the current human
|
||||
authority.
|
||||
3. **Finish the remaining ladder.** Manual verification, SHA staleness and
|
||||
reverification, human-decision mismatch, then a real replan with old-plan
|
||||
provenance.
|
||||
4. **Use the capability table above as backend work discovery.** Do not add fake
|
||||
controls. Each disabled action is concrete evidence of a missing capability.
|
||||
5. **Continue the debt slices independently.** Durable worker observations and
|
||||
manual-intervention events are the next evidence gaps. Not automatic
|
||||
maintenance yet.
|
||||
|
||||
## What the tag means
|
||||
|
||||
```text
|
||||
orchestra-release-v1 -> 34f3c28
|
||||
deployed, settled lifecycle plus the truth-detector UI baseline
|
||||
|
||||
later HEADs
|
||||
experimental plan, debt and runtime work that must earn their own
|
||||
release proof
|
||||
```
|
||||
|
||||
Use that distinction. A defect found in experimental work is not a reason to
|
||||
reopen settled architecture.
|
||||
|
||||
## The roadmap the operator set
|
||||
|
||||
```text
|
||||
A. runtime correctness finish the plan-machinery live proof
|
||||
B. evidence and debt durable worker observations, manual-intervention
|
||||
events, then rerun the projection
|
||||
C. operator surface first-class HumanDecision write API,
|
||||
effective project-config read API,
|
||||
session and context health projection
|
||||
D. adapter fix opencode activity and session resolution
|
||||
E. UI wire capabilities as backend support becomes real
|
||||
```
|
||||
|
||||
The framing that ties them together is the thing to keep. The UI says what
|
||||
Orchestra cannot expose or control. The debt ledger says which of those
|
||||
shortcomings repeatedly costs something. The burn-in says which runtime
|
||||
semantics are reliable. Those three decide what gets built next.
|
||||
|
||||
One design question to settle before B's slice two writes any code. The worker
|
||||
ring is bounded and lossy by construction. Ingesting it durably means deciding
|
||||
whether the coordinator stores every observation as an event, or only
|
||||
transitions. Storing every heartbeat's ring would write the same 41-count
|
||||
observation hundreds of times.
|
||||
@@ -0,0 +1,17 @@
|
||||
# orchestra ui mockups
|
||||
|
||||
`final/` contains the nine accepted/current screen directions:
|
||||
|
||||
1. dashboard
|
||||
2. task detail
|
||||
3. terminal live pane
|
||||
4. tasks
|
||||
5. decisions
|
||||
6. workers
|
||||
7. projects
|
||||
8. review
|
||||
9. settings
|
||||
|
||||
`iterations/` contains every generated mockup from the design session, including superseded variants.
|
||||
|
||||
`orchestra-ui-spec.md` is the accompanying implementation/design specification.
|
||||
@@ -0,0 +1,110 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"orchestra/internal/domain"
|
||||
"orchestra/internal/federation"
|
||||
"orchestra/internal/herdr"
|
||||
)
|
||||
|
||||
// F62. Run 16: the agent was asked to hand off, never wrote HANDOFF.md,
|
||||
// renewals stopped, and the lease died as ordinary idleness. Waiting is now
|
||||
// bounded: re-ask once, then give the task up with a class that says why.
|
||||
func TestUnansweredHandoffIsRetriedThenGivenUp(t *testing.T) {
|
||||
var nack map[string]any
|
||||
w, backend, _, done := phaseWorker(t, func(rw http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/nack") {
|
||||
_ = json.NewDecoder(r.Body).Decode(&nack)
|
||||
}
|
||||
rw.Write([]byte(`{}`))
|
||||
})
|
||||
defer done()
|
||||
ctx := context.Background()
|
||||
requested := func(ago time.Duration) herdr.Session {
|
||||
s := w.sessions["task"]
|
||||
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
|
||||
s.HandoffRequestedAt = time.Now().UTC().Add(-ago)
|
||||
w.sessions["task"] = s
|
||||
return s
|
||||
}
|
||||
|
||||
// Still inside the answering window: nothing said, nothing given up.
|
||||
if s, gaveUp := w.watchHandoff(ctx, "task", requested(time.Minute)); gaveUp || s.HandoffRetried {
|
||||
t.Fatalf("gave up while still waiting: gaveUp=%v session=%+v", gaveUp, s)
|
||||
}
|
||||
if len(backend.prompts) != 0 {
|
||||
t.Fatalf("re-asked too early: %q", backend.prompts)
|
||||
}
|
||||
|
||||
// Past the retry point: asked again, exactly once.
|
||||
s, gaveUp := w.watchHandoff(ctx, "task", requested(handoffRetryAfter+time.Minute))
|
||||
if gaveUp || !s.HandoffRetried || len(backend.prompts) != 1 {
|
||||
t.Fatalf("retry: gaveUp=%v retried=%v prompts=%q", gaveUp, s.HandoffRetried, backend.prompts)
|
||||
}
|
||||
if _, gaveUp = w.watchHandoff(ctx, "task", s); gaveUp || len(backend.prompts) != 1 {
|
||||
t.Fatalf("re-asked every tick: %q", backend.prompts)
|
||||
}
|
||||
|
||||
// Past the bound: a causal reclaim, and no lease left to renew.
|
||||
s = requested(handoffAnswerTimeout + time.Second)
|
||||
s.HandoffRetried = true
|
||||
w.sessions["task"] = s
|
||||
if _, gaveUp = w.watchHandoff(ctx, "task", s); !gaveUp {
|
||||
t.Fatal("an unanswered handoff waited forever")
|
||||
}
|
||||
if nack["failure_class"] != "handoff_unanswered" {
|
||||
t.Fatalf("nack = %+v", nack)
|
||||
}
|
||||
if detail, _ := nack["last_error"].(string); !strings.Contains(detail, "phase_changed") {
|
||||
t.Fatalf("the reclaim does not name the request: %q", detail)
|
||||
}
|
||||
if _, held := w.leases["task"]; held {
|
||||
t.Fatal("the given-up task kept its lease")
|
||||
}
|
||||
}
|
||||
|
||||
// The lease must survive the wait it was asked to make: an idle pane is the
|
||||
// answer Orchestra requested, not evidence of an agent that stopped working.
|
||||
func TestWaitingForAHandoffKeepsTheLease(t *testing.T) {
|
||||
renewals := 0
|
||||
api := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
|
||||
renewals++
|
||||
rw.Write([]byte(`{}`))
|
||||
}))
|
||||
defer api.Close()
|
||||
backend := &recordingBackend{status: "idle", progress: "same screen"}
|
||||
w := &worker{
|
||||
api: federation.Client{BaseURL: api.URL, WorkerID: "h", Token: "t"},
|
||||
backend: backend,
|
||||
harness: "claude",
|
||||
sessions: map[string]herdr.Session{"task": {PaneID: "pane", HandoffRequested: true, HandoffRequestedAt: time.Now().UTC()}},
|
||||
leases: map[string]lease{"task": {Epoch: "e", Version: 1, Until: time.Now(), ProgressSHA: domain.Hash([]byte("same screen"))}},
|
||||
quarantined: map[string]bool{},
|
||||
statePath: filepath.Join(t.TempDir(), "state.json"),
|
||||
}
|
||||
w.renewLeases(context.Background())
|
||||
if renewals != 1 {
|
||||
t.Fatalf("a lease waiting on a requested handoff renewed %d times, want 1", renewals)
|
||||
}
|
||||
|
||||
// Past the bound the exemption stops: watchHandoff has given the task up
|
||||
// by then, and nothing keeps an unanswered request alive.
|
||||
s := w.sessions["task"]
|
||||
s.HandoffRequestedAt = time.Now().UTC().Add(-handoffAnswerTimeout - time.Second)
|
||||
w.sessions["task"] = s
|
||||
l := w.leases["task"]
|
||||
l.Until = time.Now()
|
||||
w.leases["task"] = l
|
||||
w.renewLeases(context.Background())
|
||||
if renewals != 1 {
|
||||
t.Fatalf("the exemption outlived its bound: renewals=%d", renewals)
|
||||
}
|
||||
}
|
||||
@@ -174,6 +174,7 @@ func releaseBackoff(attempts int) time.Duration {
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
type projectConfig struct {
|
||||
Repo string `json:"repo"`
|
||||
Root string `json:"worktree_root"`
|
||||
@@ -673,6 +674,13 @@ func (w *worker) releaseReady(ctx context.Context) {
|
||||
w.advanceRelease(ctx, id, s)
|
||||
continue
|
||||
}
|
||||
if s.HandoffRequested {
|
||||
next, gaveUp := w.watchHandoff(ctx, id, s)
|
||||
if gaveUp {
|
||||
continue
|
||||
}
|
||||
s = next
|
||||
}
|
||||
w.rotationTick(ctx, id, s)
|
||||
}
|
||||
}
|
||||
@@ -772,7 +780,7 @@ func (w *worker) rotationTick(ctx context.Context, id string, s herdr.Session) {
|
||||
w.recordError(fmt.Errorf("rotation %s threshold prompt: %w", id, err))
|
||||
return
|
||||
}
|
||||
s.HandoffRequested, s.HandoffReason = true, d.Reason
|
||||
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, d.Reason, time.Now().UTC()
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
}
|
||||
@@ -1166,6 +1174,70 @@ func (w *worker) submit(ctx context.Context, id string, s herdr.Session, e compl
|
||||
// the directory also holds for a worktree that has no inner .gitignore.
|
||||
const stageExclude = ":!.orchestra"
|
||||
|
||||
// F62: a requested handoff nobody answers was invisible. Renewals stopped,
|
||||
// the lease expired, and the task lost an attempt with nothing on record
|
||||
// saying a handoff had ever been asked for — worker health showed only "agent
|
||||
// status idle and pane unchanged", 34 times in run 16.
|
||||
const (
|
||||
handoffRetryAfter = 4 * time.Minute
|
||||
handoffAnswerTimeout = 10 * time.Minute
|
||||
)
|
||||
|
||||
// watchHandoff bounds the wait for an agent's handoff answer: re-send the
|
||||
// request once, then give the task up with a class that names the cause. It
|
||||
// returns the session to keep using and whether the task was given up.
|
||||
func (w *worker) watchHandoff(ctx context.Context, id string, s herdr.Session) (herdr.Session, bool) {
|
||||
if s.HandoffRequestedAt.IsZero() {
|
||||
// A session persisted before the stamp existed, or requested by a path
|
||||
// that does not set it. Start the clock now rather than time out a
|
||||
// request retroactively.
|
||||
s.HandoffRequestedAt = time.Now().UTC()
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
return s, false
|
||||
}
|
||||
waited := time.Since(s.HandoffRequestedAt)
|
||||
if waited < handoffRetryAfter {
|
||||
return s, false
|
||||
}
|
||||
if waited < handoffAnswerTimeout {
|
||||
if s.HandoffRetried || w.executionBackend() == nil {
|
||||
return s, false
|
||||
}
|
||||
a := herdr.CLIAdapter{Backend: w.executionBackend(), Harness: w.harness}
|
||||
var err error
|
||||
if s.HandoffReason != "" {
|
||||
err = a.RequestHandoffReason(ctx, s, s.HandoffReason, nil)
|
||||
} else {
|
||||
err = a.RequestHandoff(ctx, s)
|
||||
}
|
||||
if err != nil {
|
||||
w.recordError(fmt.Errorf("handoff %s re-request: %w", id, err))
|
||||
return s, false
|
||||
}
|
||||
s.HandoffRetried = true
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
w.recordError(fmt.Errorf("handoff %s (%s) unanswered for %s: request re-sent", id, s.HandoffReason, waited.Round(time.Second)))
|
||||
return s, false
|
||||
}
|
||||
l, ok := w.leases[id]
|
||||
if !ok {
|
||||
return s, false
|
||||
}
|
||||
detail := fmt.Sprintf("handoff requested (%s) and unanswered for %s", s.HandoffReason, waited.Round(time.Second))
|
||||
if err := w.api.NackStart(ctx, id, l.Epoch, l.Version, "handoff_unanswered", detail, w.sessionEvidence(ctx, id, s)); err != nil {
|
||||
w.recordError(fmt.Errorf("handoff timeout %s: %w", id, err))
|
||||
return s, false
|
||||
}
|
||||
// The coordinator answers with TaskReleased; its replay quarantines the
|
||||
// pane. Drop the lease here so nothing renews it in the meantime.
|
||||
delete(w.leases, id)
|
||||
_ = w.save()
|
||||
w.recordError(errors.New(detail))
|
||||
return s, true
|
||||
}
|
||||
|
||||
func (w *worker) renewLeases(ctx context.Context) {
|
||||
if w.executionBackend() == nil {
|
||||
return
|
||||
@@ -1199,6 +1271,11 @@ func (w *worker) renewLeases(ctx context.Context) {
|
||||
case l.ProgressSHA == "":
|
||||
// First renewal has no baseline to compare against. Record one and
|
||||
// allow this renewal; the next one must show real movement.
|
||||
case s.HandoffRequested && !s.HandoffRequestedAt.IsZero() && time.Since(s.HandoffRequestedAt) < handoffAnswerTimeout:
|
||||
// Orchestra told this agent to stop and write its handoff. A quiet
|
||||
// pane is the answer it was asked for, so the lease is held while
|
||||
// the wait is explicitly bounded (F62). Past the bound the case
|
||||
// stops matching and watchHandoff has already given the task up.
|
||||
default:
|
||||
w.recordError(fmt.Errorf("lease %s not renewed: agent status %s and pane unchanged since the last renewal", taskID, status))
|
||||
continue
|
||||
@@ -1932,7 +2009,7 @@ func (w *worker) federatedTurn(ctx context.Context, id string, a herdr.Adapter,
|
||||
w.recordError(fmt.Errorf("reconcile failure handoff %s: %w", id, err))
|
||||
return
|
||||
}
|
||||
s.HandoffRequested, s.HandoffReason = true, "reconcile_failure"
|
||||
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "reconcile_failure", time.Now().UTC()
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
return
|
||||
@@ -2009,7 +2086,7 @@ func (w *worker) rotateForPhase(ctx context.Context, id string, a herdr.Adapter,
|
||||
w.recordError(fmt.Errorf("phase rotation %s: %w", id, err))
|
||||
return
|
||||
}
|
||||
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
|
||||
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "phase_changed", time.Now().UTC()
|
||||
w.sessions[id] = s
|
||||
_ = w.save()
|
||||
log.Printf("phase changed for %s: session rotating", id)
|
||||
|
||||
@@ -1589,6 +1589,12 @@ func main() {
|
||||
case "invalid_handoff":
|
||||
typ = "TaskBlocked"
|
||||
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonHandoffValidation), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_nacked", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
||||
case "handoff_unanswered":
|
||||
// F62: not a launch failure. The agent was asked to hand off
|
||||
// and never did, so the reclaim says exactly that instead of
|
||||
// arriving as an ordinary idle expiry.
|
||||
typ = "TaskReleased"
|
||||
p, _ = json.Marshal(map[string]any{"reason": "handoff_unanswered", "failure_class": b.FailureClass, "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "handoff_unanswered", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
||||
case "launch_uncertain":
|
||||
typ = "TaskNeedsAttention"
|
||||
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonLeaseFailure), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_uncertain", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
|
||||
@@ -1825,6 +1831,25 @@ func main() {
|
||||
coordinator.ReconcileFailureHandoff = v
|
||||
}
|
||||
}
|
||||
// The third reconciliation point, and the same reasoning the submitted
|
||||
// work loop above already uses: a blocked task cannot be leased, so a
|
||||
// pre-lease hook can never see the reply that should unblock it (F64).
|
||||
// Orchestra stopped and asked the human a question; nothing else here
|
||||
// will go and read the answer.
|
||||
go func() {
|
||||
ticker := time.NewTicker(time.Minute)
|
||||
defer ticker.Stop()
|
||||
for range ticker.C {
|
||||
for _, t := range s.Tasks() {
|
||||
if t.State != domain.StateBlocked || !domain.BlockReasonAwaitsReply(t.BlockReason) {
|
||||
continue
|
||||
}
|
||||
if err := reconciler.Reconcile(context.Background(), t.ID); err != nil {
|
||||
log.Printf("reconcile blocked task %s: %v", t.ID, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
if path := os.Getenv("ORCHESTRA_JSONL"); path != "" {
|
||||
sup := &provider.Supervisor{Name: "jsonl", Run: func(ctx context.Context) error {
|
||||
|
||||
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
@@ -251,7 +251,7 @@ func DebtClassForBlockReason(r BlockReason) (DebtClass, bool) {
|
||||
// classes a worker actually emits are listed; an unknown one is not guessed at.
|
||||
func DebtClassForFailureClass(f string) (DebtClass, bool) {
|
||||
switch f {
|
||||
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired":
|
||||
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired", "handoff_unanswered":
|
||||
return DebtOperational, true
|
||||
case "invalid_handoff":
|
||||
return DebtCorrectness, true
|
||||
|
||||
@@ -82,6 +82,23 @@ const (
|
||||
BlockReasonUnknown BlockReason = "unknown"
|
||||
)
|
||||
|
||||
// BlockReasonAwaitsReply reports whether this stop is a question waiting on a
|
||||
// human, rather than a fault or a budget. Every reason listed here resumes on
|
||||
// a reply, so both the loop that goes and reads replies and the loop that
|
||||
// returns answered tasks to the queue must agree on the set. They did not:
|
||||
// plan_mismatch blocked for an answer that nothing ever went to fetch, and
|
||||
// nothing would have resumed it if it had (F64).
|
||||
//
|
||||
// operator_required is deliberately absent. A task that has spent its question
|
||||
// budget continues when an operator says so, not when someone replies.
|
||||
func BlockReasonAwaitsReply(r BlockReason) bool {
|
||||
switch r {
|
||||
case BlockReasonHumanDecision, BlockReasonTrajectoryGate, BlockReasonPlanMismatch:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (r BlockReason) Valid() bool {
|
||||
switch r {
|
||||
case BlockReasonLeaseFailure, BlockReasonWorkerOffline, BlockReasonLeaseExpired,
|
||||
@@ -168,19 +185,19 @@ type Task struct {
|
||||
QualityGate string `json:"quality_gate,omitempty"`
|
||||
// Block evidence is projected from TaskBlocked so terminal records remain
|
||||
// diagnosable after the live coordinator mapping is gone.
|
||||
Blocker string `json:"blocker,omitempty"`
|
||||
BlockReason BlockReason `json:"block_reason,omitempty"`
|
||||
BlockedAt time.Time `json:"blocked_at,omitempty"`
|
||||
LastPaneID string `json:"last_pane_id,omitempty"`
|
||||
LastHarness string `json:"last_harness_id,omitempty"`
|
||||
Blocker string `json:"blocker,omitempty"`
|
||||
BlockReason BlockReason `json:"block_reason,omitempty"`
|
||||
BlockedAt time.Time `json:"blocked_at,omitempty"`
|
||||
LastPaneID string `json:"last_pane_id,omitempty"`
|
||||
LastHarness string `json:"last_harness_id,omitempty"`
|
||||
// LastLeaseEpoch is the fencing token of the lease that most recently
|
||||
// ended. A worker can push its release anchor and only then discover the
|
||||
// lease expired; the finished work is durable in git but the commit can
|
||||
// never land. Retaining the epoch lets exactly that owner still commit
|
||||
// while the task sits unleased.
|
||||
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
|
||||
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
|
||||
LastSession SessionEvidence `json:"last_session,omitempty"`
|
||||
LastLeaseEpoch string `json:"last_lease_epoch,omitempty"`
|
||||
PaneState string `json:"pane_state,omitempty"` // open, closed, unreachable, unknown
|
||||
LastSession SessionEvidence `json:"last_session,omitempty"`
|
||||
// Recovery state is part of the durable projection, never process-local
|
||||
// router memory. This makes retry and operator diagnostics survive a
|
||||
// coordinator restart.
|
||||
@@ -217,7 +234,7 @@ type Task struct {
|
||||
// current one, oldest first. A superseded plan stays queryable: the
|
||||
// verification recorded against it is provenance, not garbage.
|
||||
PlanHistory []string `json:"plan_history,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
// ReviewRef binds a sealed review artifact to one commit.
|
||||
|
||||
@@ -55,6 +55,13 @@ type PlanPhaseRecord struct {
|
||||
// EvidenceRef is the CAS ref of the captured command output.
|
||||
EvidenceRef string `json:"evidence_ref,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
// ManualAtSHA is the tree a human was actually looking at when they signed
|
||||
// this phase off. A manual check on most projects is a human reading
|
||||
// output, so a sign-off establishes something about one tree and nothing
|
||||
// about the next one (F63). Rerunning the automated half re-establishes it
|
||||
// at the new commit; the manual half has to be given again, and this is
|
||||
// what makes the difference visible instead of assumed.
|
||||
ManualAtSHA string `json:"manual_at_sha,omitempty"`
|
||||
}
|
||||
|
||||
// Stale reports whether the tree has moved since this phase was verified. A
|
||||
|
||||
@@ -174,6 +174,14 @@ type Session struct {
|
||||
// its §6.1 handoff (HandoffFile) — avoids re-sending the same prompt
|
||||
// every tick while Release keeps waiting for the file to appear.
|
||||
HandoffRequested bool `json:"handoff_requested,omitempty"`
|
||||
// HandoffRequestedAt stamps that prompt. A request nobody answers used to
|
||||
// end as an ordinary idle expiry, indistinguishable from an agent that
|
||||
// never started (F62); the stamp is what makes the wait bounded and the
|
||||
// giving-up causal.
|
||||
HandoffRequestedAt time.Time `json:"handoff_requested_at,omitempty"`
|
||||
// HandoffRetried records that the request was re-sent once, so a session
|
||||
// waiting on an answer is not re-prompted every tick.
|
||||
HandoffRetried bool `json:"handoff_retried,omitempty"`
|
||||
// HandoffReason is selected by the coordinator when it asks for the
|
||||
// semantic report. The checkout worker, rather than the harness, copies
|
||||
// it into the canonical handoff it seals at release time.
|
||||
|
||||
@@ -58,18 +58,27 @@ func RequestHumanDecision(s *store.Store, project registry.Project, taskID strin
|
||||
return blockTask(s, t, domain.BlockReasonHumanDecision, req.Render(), &req)
|
||||
}
|
||||
|
||||
// fenceToLease binds a lifecycle event to the lease that is producing it.
|
||||
// Store.Append fences every lifecycle event on a leased task against the
|
||||
// current owner and epoch, so an event that omits them is a conflict rather
|
||||
// than a block. Every coordinator-side stop goes through here: two of them did
|
||||
// not, and both failed silently against a live lease (F65). A task stops for a
|
||||
// human only while some session is running, so the leased case is the only one
|
||||
// that ever mattered.
|
||||
func fenceToLease(payload map[string]any, t domain.Task) {
|
||||
if t.Lease == nil {
|
||||
return
|
||||
}
|
||||
payload["harness_id"] = t.Lease.HarnessID
|
||||
payload["lease_epoch"] = t.Lease.Epoch
|
||||
}
|
||||
|
||||
func blockTask(s *store.Store, t domain.Task, reason domain.BlockReason, blocker string, req *domain.DecisionRequest) (domain.Event, error) {
|
||||
payload := map[string]any{
|
||||
"blocker": blocker, "block_reason": string(reason),
|
||||
"lifecycle_phase": "awaiting_human",
|
||||
}
|
||||
if t.Lease != nil {
|
||||
// Store.Append fences every lifecycle event on a leased task against
|
||||
// the current owner and epoch. A question from a session that no
|
||||
// longer owns the task is a conflict, not a block.
|
||||
payload["harness_id"] = t.Lease.HarnessID
|
||||
payload["lease_epoch"] = t.Lease.Epoch
|
||||
}
|
||||
fenceToLease(payload, t)
|
||||
if req != nil {
|
||||
payload["decision_request"] = req
|
||||
}
|
||||
@@ -107,11 +116,9 @@ func ResumeAnsweredBlockers(s *store.Store) ([]domain.Event, error) {
|
||||
if t.State != domain.StateBlocked {
|
||||
continue
|
||||
}
|
||||
switch t.BlockReason {
|
||||
case domain.BlockReasonHumanDecision, domain.BlockReasonTrajectoryGate:
|
||||
default:
|
||||
// operator_required is deliberately not resumed by a reply. An
|
||||
// operator decides when a task that spent its budget continues.
|
||||
// operator_required is deliberately not resumed by a reply. An
|
||||
// operator decides when a task that spent its budget continues.
|
||||
if !domain.BlockReasonAwaitsReply(t.BlockReason) {
|
||||
continue
|
||||
}
|
||||
if !blockerAnswered(s, t.ID, t.BlockReason) {
|
||||
|
||||
@@ -154,11 +154,13 @@ func blockForPlanMismatch(s *store.Store, taskID string, m domain.PlanMismatch)
|
||||
packet += "- evidence: " + oneLine(e) + "\n"
|
||||
}
|
||||
packet += "\nReply to say how to proceed. Your reply becomes a recorded decision and outranks the plan. If it resolves the contradiction, the task resumes on the same plan; say so explicitly if you want the plan rewritten instead.\n"
|
||||
b, err := json.Marshal(map[string]any{
|
||||
payload := map[string]any{
|
||||
"blocker": packet,
|
||||
"block_reason": string(domain.BlockReasonPlanMismatch),
|
||||
"lifecycle_phase": "awaiting_human",
|
||||
})
|
||||
}
|
||||
fenceToLease(payload, t)
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -215,7 +215,16 @@ func TestHumanAnswerResumesTheSamePlanWithoutResealing(t *testing.T) {
|
||||
if !PlanMismatchAnswered(s, id) {
|
||||
t.Fatal("the human answered and the task is still waiting")
|
||||
}
|
||||
// F64: asserting the predicate is not asserting the resume. This test
|
||||
// passed for as long as the predicate had no caller, while a task blocked
|
||||
// on a mismatch stayed blocked forever however the human replied.
|
||||
if events, err := ResumeAnsweredBlockers(s); err != nil || len(events) != 1 {
|
||||
t.Fatalf("an answered mismatch did not return to the queue: events=%v err=%v", events, err)
|
||||
}
|
||||
after, _ := s.Task(id)
|
||||
if after.State != domain.StateQueued {
|
||||
t.Fatalf("state = %s after the human answered, want queued", after.State)
|
||||
}
|
||||
if after.PlanRef != planRef {
|
||||
t.Fatal("answering the question replaced the plan")
|
||||
}
|
||||
|
||||
@@ -124,7 +124,12 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
|
||||
// was established.
|
||||
record.Status = domain.PlanPhaseAwaitingManual
|
||||
}
|
||||
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID) {
|
||||
// Carry the confirmed tree forward as provenance. Without it a second
|
||||
// rerun would compare against nothing and re-inherit the sign-off.
|
||||
if prior, ok := t.PlanPhase(phaseID); ok {
|
||||
record.ManualAtSHA = prior.ManualAtSHA
|
||||
}
|
||||
if record.Status != domain.PlanPhaseInProgress && manuallySignedOff(s, t, phaseID, record) {
|
||||
record.Status = domain.PlanPhaseVerified
|
||||
}
|
||||
if ref, err := s.PutArtifact(verificationEvidence(runs)); err == nil {
|
||||
@@ -135,7 +140,7 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
|
||||
payload := map[string]any{
|
||||
"plan_ref": record.PlanRef, "phase_id": record.PhaseID, "status": string(record.Status),
|
||||
"commands": record.Commands, "exit_codes": record.ExitCodes, "at_sha": record.AtSHA,
|
||||
"evidence_ref": record.EvidenceRef, "at": record.At,
|
||||
"evidence_ref": record.EvidenceRef, "at": record.At, "manual_at_sha": record.ManualAtSHA,
|
||||
}
|
||||
if t.Lease != nil {
|
||||
payload["harness_id"], payload["lease_epoch"] = t.Lease.HarnessID, t.Lease.Epoch
|
||||
@@ -149,9 +154,20 @@ func RecordPlanPhaseVerification(s *store.Store, project registry.Project, taskI
|
||||
}
|
||||
|
||||
// manuallySignedOff reports whether a human has already approved this exact
|
||||
// phase of this exact plan. The subject carries both, so a later "looks good"
|
||||
// on an unrelated thread cannot satisfy a gate nobody was discussing.
|
||||
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string) bool {
|
||||
// phase of this exact plan, against the tree this run examined. The subject
|
||||
// carries plan and phase, so a later "looks good" on an unrelated thread
|
||||
// cannot satisfy a gate nobody was discussing.
|
||||
//
|
||||
// The tree matters as much as the subject (F63). A sign-off is a human saying
|
||||
// they read what this code prints; an edit afterwards can change exactly that.
|
||||
// A record whose ManualAtSHA names a different commit is therefore not signed
|
||||
// off, and waits for the human again. A sign-off given before any run has no
|
||||
// confirmed tree to compare against and still counts, which keeps the ordinary
|
||||
// ordering unchanged.
|
||||
func manuallySignedOff(s *store.Store, t domain.Task, phaseID string, record domain.PlanPhaseRecord) bool {
|
||||
if record.ManualAtSHA != "" && record.ManualAtSHA != record.AtSHA {
|
||||
return false
|
||||
}
|
||||
intent, err := s.EffectiveIntent(t.ID)
|
||||
if err != nil {
|
||||
return false
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
|
||||
const shaOne = "1111111111111111111111111111111111111111"
|
||||
const shaTwo = "2222222222222222222222222222222222222222"
|
||||
const shaThree = "3333333333333333333333333333333333333333"
|
||||
|
||||
func planProject() registry.Project {
|
||||
p := registry.Project{
|
||||
@@ -27,6 +28,11 @@ func planProject() registry.Project {
|
||||
func planWith(t *testing.T, markdown string) (*store.Store, registry.Project, string) {
|
||||
t.Helper()
|
||||
s, id := phaseStore(t)
|
||||
// Leased, because everything these tests drive comes from a live implement
|
||||
// session. Skipping it hid F65: two coordinator-side stops omitted the
|
||||
// fencing fields Store.Append requires on a leased task, and every test
|
||||
// passed because no test ever leased one.
|
||||
lease(t, s, id)
|
||||
project := planProject()
|
||||
if _, err := AdvanceWorkPhase(s, project, id, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -251,6 +257,14 @@ func TestLegacyPlanIsExplicitlyNonProgressable(t *testing.T) {
|
||||
// signOff records a human decision bound to one phase of one plan, which is
|
||||
// the only thing that satisfies a manual verification gate.
|
||||
func signOff(t *testing.T, s *store.Store, taskID, subject string) {
|
||||
t.Helper()
|
||||
signOffFrom(t, s, taskID, subject, "signoff-"+subject)
|
||||
}
|
||||
|
||||
// signOffFrom names the comment the sign-off came from. Two sign-offs on one
|
||||
// subject are a real sequence once a rerun sends a phase back to the human,
|
||||
// and provenance is unique per comment.
|
||||
func signOffFrom(t *testing.T, s *store.Store, taskID, subject, externalID string) {
|
||||
t.Helper()
|
||||
task, _ := s.Task(taskID)
|
||||
if err := s.Append(domain.Event{
|
||||
@@ -259,9 +273,60 @@ func signOff(t *testing.T, s *store.Store, taskID, subject string) {
|
||||
Payload: mustJSONBytes(t, map[string]any{
|
||||
"decision_id": domain.NewID(), "kind": "answer", "subject": subject,
|
||||
"value": "manual steps confirmed",
|
||||
"source": map[string]any{"provider": "gitea", "external_id": "signoff-" + subject},
|
||||
"source": map[string]any{"provider": "gitea", "external_id": externalID},
|
||||
}),
|
||||
}); err != nil {
|
||||
t.Fatalf("sign off: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// F63, found live on run 19. A manual sign-off says a human read what this
|
||||
// code prints. An edit afterwards can change exactly that, so rerunning the
|
||||
// automated half at a new commit must not carry the human half with it.
|
||||
func TestASignOffDoesNotSurviveTheTreeItWasGivenAgainst(t *testing.T) {
|
||||
s, project, id := planWith(t, twoPhasePlan)
|
||||
run := []VerificationRun{{Command: []string{"go", "test", "./internal/..."}, ExitCode: 0}}
|
||||
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaOne, run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ := s.Task(id)
|
||||
signOff(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"))
|
||||
task, _ = s.Task(id)
|
||||
rec, _ := task.PlanPhase("phase-2")
|
||||
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaOne {
|
||||
t.Fatalf("sign-off did not bind to the tree it read: %+v", rec)
|
||||
}
|
||||
|
||||
// The tree moves and the phase is re-verified. The commands pass again;
|
||||
// the human has not seen the new output.
|
||||
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaTwo, run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ = s.Task(id)
|
||||
rec, _ = task.PlanPhase("phase-2")
|
||||
if rec.Status != domain.PlanPhaseAwaitingManual {
|
||||
t.Fatalf("status = %q at a tree the human never saw, want awaiting_manual_verification", rec.Status)
|
||||
}
|
||||
if rec.ManualAtSHA != shaOne {
|
||||
t.Fatalf("the confirmed tree was lost: %+v", rec)
|
||||
}
|
||||
|
||||
// A second rerun must not re-inherit it either, which is what carrying
|
||||
// ManualAtSHA forward is for.
|
||||
if _, err := RecordPlanPhaseVerification(s, project, id, "phase-2", shaThree, run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ = s.Task(id)
|
||||
rec, _ = task.PlanPhase("phase-2")
|
||||
if rec.Status != domain.PlanPhaseAwaitingManual {
|
||||
t.Fatalf("a second rerun re-inherited the sign-off: %q", rec.Status)
|
||||
}
|
||||
|
||||
// Signing off again, on the tree that is now current, verifies it.
|
||||
signOffFrom(t, s, id, domain.PlanPhaseSubject(task.PlanRef, "phase-2"), "signoff-second")
|
||||
task, _ = s.Task(id)
|
||||
rec, _ = task.PlanPhase("phase-2")
|
||||
if rec.Status != domain.PlanPhaseVerified || rec.ManualAtSHA != shaThree {
|
||||
t.Fatalf("a fresh sign-off did not verify the current tree: %+v", rec)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -67,11 +67,13 @@ func raiseTrajectoryGate(s *store.Store, t domain.Task, from, to domain.WorkPhas
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := json.Marshal(map[string]any{
|
||||
payload := map[string]any{
|
||||
"blocker": packet,
|
||||
"block_reason": string(domain.BlockReasonTrajectoryGate),
|
||||
"lifecycle_phase": "awaiting_human",
|
||||
})
|
||||
}
|
||||
fenceToLease(payload, t)
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -272,6 +272,9 @@ func (s *Store) apply(e domain.Event) error {
|
||||
}
|
||||
if p.Subject == domain.PlanPhaseSubject(rec.PlanRef, rec.PhaseID) {
|
||||
t.PlanProgress.Phases[i].Status = domain.PlanPhaseVerified
|
||||
// Record which tree the sign-off was about, so a later run
|
||||
// at a different commit cannot inherit it (F63).
|
||||
t.PlanProgress.Phases[i].ManualAtSHA = rec.AtSHA
|
||||
t.Version = e.Version
|
||||
s.replaceTask(e.TaskID, t)
|
||||
}
|
||||
|
||||
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
@@ -0,0 +1,635 @@
|
||||
# orchestra web control ui
|
||||
|
||||
status: mockup specification
|
||||
app: orchestra
|
||||
shared design system: ethos
|
||||
accent: signal violet `#8F7AE5`
|
||||
motif: routing fork
|
||||
primary operator: single infrastructure operator
|
||||
|
||||
## 1. purpose
|
||||
|
||||
orchestra's web ui is an operator console for understanding and steering agent work without reconstructing state from logs, tmux sessions, gitea, or the event stream.
|
||||
|
||||
it has three jobs:
|
||||
|
||||
1. show what needs the operator now.
|
||||
2. show what orchestra believes and what an agent is actually doing.
|
||||
3. expose intervention and forensic detail without making the default screens noisy.
|
||||
|
||||
it is not a generic project-management product, an analytics dashboard, or a chat-first interface.
|
||||
|
||||
## 2. ethos application
|
||||
|
||||
orchestra inherits the ethos shell and neutral system unchanged.
|
||||
|
||||
### fingerprint
|
||||
|
||||
- accent: signal violet `#8F7AE5`
|
||||
- motif: routing fork
|
||||
- accent use: active navigation, focus, selected phase, primary operator action, routing/transition marks
|
||||
- motif use: app mark, phase transitions, routing state, empty states, one hero/detail moment
|
||||
|
||||
suggested token block:
|
||||
|
||||
```css
|
||||
[data-app="orchestra"] {
|
||||
--accent: #8F7AE5;
|
||||
--accent-hi: #A291EC;
|
||||
--accent-dim: rgba(143, 122, 229, 0.14);
|
||||
--accent-line: rgba(143, 122, 229, 0.32);
|
||||
--accent-glow: rgba(143, 122, 229, 0.22);
|
||||
}
|
||||
```
|
||||
|
||||
### type
|
||||
|
||||
- geist sans: labels, task names, explanations, buttons, headings, operator-authored prose
|
||||
- geist mono: task ids, epochs, worker ids, pane ids, timestamps, percentages, context counts, revisions, paths, refs, sha values, lease durations
|
||||
- machine values must never silently use sans for visual convenience
|
||||
|
||||
### surfaces
|
||||
|
||||
- page room: `--bg-0`
|
||||
- normal panel: `--bg-1`
|
||||
- selected/raised content: `--bg-2`
|
||||
- hover: `--bg-3`
|
||||
- borders: 1px `--line`, `--line-hi` only where selection or risk needs stronger separation
|
||||
- no blur or glass
|
||||
- accent never washes a whole content panel
|
||||
|
||||
### motion
|
||||
|
||||
all ui transitions use ethos mechanical motion: 130–170 ms, `cubic-bezier(0.2, 0, 0, 1)`, no spring or bounce.
|
||||
|
||||
## 3. shared shell
|
||||
|
||||
### desktop
|
||||
|
||||
- 64px vertical icon rail
|
||||
- 56px top bar
|
||||
- main content fills remaining viewport
|
||||
- global `cmd+k` command/search surface in the top bar
|
||||
- right side of top bar shows only high-value machine state: orchestra revision, live state, current time
|
||||
|
||||
primary navigation:
|
||||
|
||||
1. dashboard
|
||||
2. tasks
|
||||
3. decisions
|
||||
4. workers
|
||||
5. projects
|
||||
6. review
|
||||
7. settings
|
||||
|
||||
terminal is entered from a task or worker and does not need permanent primary navigation.
|
||||
|
||||
### mobile
|
||||
|
||||
- rail becomes bottom navigation with no more than five primary items
|
||||
- secondary pages move behind task/project drill-down or header menus
|
||||
- machine readouts relocate into detail screens rather than disappearing
|
||||
- terminal remains a dedicated full-screen view
|
||||
|
||||
## 4. information hierarchy
|
||||
|
||||
orchestra should visually distinguish five classes of information.
|
||||
|
||||
| class | meaning | treatment |
|
||||
|---|---|---|
|
||||
| operator authority | human decision, correction, constraint | high prominence, accent signal |
|
||||
| verified orchestra state | lifecycle, lease, phase, review, refs | neutral surfaces, machine values in mono |
|
||||
| agent claims | proposed plan, handoff text, self-reported risk | visibly labeled as agent-supplied |
|
||||
| historical/superseded | stale plan, previous handoff, old decision | muted, never visually dominant |
|
||||
| fault/attention | blocked, refused, stale, mismatch | status color + explicit reason |
|
||||
|
||||
raw events and implementation details are drill-down evidence, not default-page content.
|
||||
|
||||
## 5. dashboard
|
||||
|
||||
mockup: `final/01-dashboard.png`
|
||||
|
||||
purpose: answer "what needs me, what is running, and is the system healthy?"
|
||||
|
||||
### layout
|
||||
|
||||
three primary regions only:
|
||||
|
||||
1. **needs your attention**
|
||||
- trajectory gate
|
||||
- blocking human decision
|
||||
- pr awaiting review
|
||||
- closed-without-merge/operator-required state
|
||||
- one line explaining why each item needs the operator
|
||||
|
||||
2. **running tasks**
|
||||
- task name/id
|
||||
- current phase
|
||||
- worker/harness
|
||||
- last real progress
|
||||
- context occupancy
|
||||
- lease timing
|
||||
- compact routing-fork phase indicator
|
||||
|
||||
3. **system state and capacity**
|
||||
- workers online/available
|
||||
- quota status
|
||||
- active leases
|
||||
- router queue/rejection state
|
||||
- source reconciliation
|
||||
- herdr health
|
||||
|
||||
### rules
|
||||
|
||||
- no disk, cpu, ram, or generic host-monitor metrics unless they directly block orchestration
|
||||
- no vanity charts
|
||||
- health should show actual counts/times, not vague "running" indicators
|
||||
- dashboard actions should navigate to the relevant task/review/worker rather than becoming an operator-control surface itself
|
||||
|
||||
## 6. tasks list
|
||||
|
||||
mockup: `final/04-tasks.png`
|
||||
|
||||
purpose: browse all work under orchestra and locate a task quickly.
|
||||
|
||||
### groups
|
||||
|
||||
- needs attention
|
||||
- running
|
||||
- waiting/queued
|
||||
- completed
|
||||
- failed
|
||||
|
||||
### row fields
|
||||
|
||||
- task title/id
|
||||
- project
|
||||
- current phase
|
||||
- compact routing-fork phase path
|
||||
- worker/harness when assigned
|
||||
- last update/progress
|
||||
- context occupancy for active sessions
|
||||
- status
|
||||
|
||||
### filters
|
||||
|
||||
- project
|
||||
- phase
|
||||
- worker/harness
|
||||
- status
|
||||
- attention state
|
||||
|
||||
avoid turning this into a kanban board. orchestra's lifecycle is stateful and event-backed; the list should expose state, not encourage arbitrary drag-and-drop mutation.
|
||||
|
||||
## 7. task detail
|
||||
|
||||
mockup: `final/02-task-detail.png`
|
||||
|
||||
purpose: provide one calm page answering "what are we doing?", "what is happening?", and "where are we in the workflow?"
|
||||
|
||||
### top area
|
||||
|
||||
- title
|
||||
- task id
|
||||
- project
|
||||
- current phase
|
||||
- steer/correct primary action
|
||||
- secondary actions menu
|
||||
|
||||
### what we're doing
|
||||
|
||||
show only:
|
||||
|
||||
- goal
|
||||
- acceptance summary
|
||||
- latest effective human decisions when present
|
||||
- active constraints when present
|
||||
|
||||
full contract, all decisions, and historical versions open in drill-down views.
|
||||
|
||||
### live execution
|
||||
|
||||
show only:
|
||||
|
||||
- worker/harness
|
||||
- pane
|
||||
- lease remaining
|
||||
- context occupancy
|
||||
- last progress
|
||||
- one-line current activity
|
||||
|
||||
links:
|
||||
|
||||
- open live pane
|
||||
- worker logs
|
||||
- launch context
|
||||
- git diff
|
||||
|
||||
### workflow
|
||||
|
||||
render the ace-fca path visibly:
|
||||
|
||||
`frame → research → plan → implement → review → pr → merged`
|
||||
|
||||
completed phases are quiet green, current phase uses the orchestra accent, future phases remain neutral.
|
||||
|
||||
phase detail, artifacts, refusal history, and raw events belong in drill-downs rather than the default task screen.
|
||||
|
||||
## 8. live pane / terminal
|
||||
|
||||
mockup: `final/03-terminal-live-pane.png`
|
||||
|
||||
purpose: show the actual tmux/herdr pane, not a reconstructed log view.
|
||||
|
||||
### behavior
|
||||
|
||||
- read-only by default
|
||||
- exact pane dimensions and terminal rendering preserved when possible
|
||||
- fit mode may scale visually but must not alter the underlying pane
|
||||
- scrollback inspection must not disturb the live cursor
|
||||
- `take control` explicitly enables keyboard forwarding
|
||||
- strong visual state when control is captured
|
||||
- escape or a fixed shortcut releases control
|
||||
|
||||
### why this screen exists
|
||||
|
||||
it must reveal real tui state such as:
|
||||
|
||||
- input still sitting in the editor
|
||||
- queued input
|
||||
- paste placeholders
|
||||
- permission/approval dialogs
|
||||
- `/clear`
|
||||
- `@HANDOFF.md`
|
||||
- agent busy/idle presentation
|
||||
- tmux status line and pane identity
|
||||
|
||||
### surrounding chrome
|
||||
|
||||
keep it minimal:
|
||||
|
||||
- worker/pane identity
|
||||
- read-only/control state
|
||||
- fit/copy/scrollback controls
|
||||
- lease epoch/timing
|
||||
- context occupancy
|
||||
- last verified progress
|
||||
- pane/worker health
|
||||
- small live event strip for orchestra-originated interactions and confirmations
|
||||
|
||||
when a pane is lost, preserve the last captured frame and state the exact loss time and lease consequence.
|
||||
|
||||
## 9. decisions list
|
||||
|
||||
mockup: `final/05-decisions.png`
|
||||
|
||||
purpose: inspect durable human authority and approvals across tasks.
|
||||
|
||||
this is not the human attention queue; that is review.
|
||||
|
||||
### list fields
|
||||
|
||||
- decision id
|
||||
- short decision text
|
||||
- task
|
||||
- phase
|
||||
- source/provenance
|
||||
- requested/recorded time
|
||||
- status: active, superseded, waiting, approved, rejected, auto
|
||||
- operator/actor
|
||||
|
||||
### filters
|
||||
|
||||
- active/superseded
|
||||
- kind: decision, correction, constraint, answer, scope change
|
||||
- project
|
||||
- task
|
||||
- actor/source
|
||||
|
||||
### detail
|
||||
|
||||
opening a decision should show:
|
||||
|
||||
- exact value
|
||||
- subject
|
||||
- provenance
|
||||
- supersedes/superseded-by links
|
||||
- event sequence
|
||||
- effective authority impact
|
||||
|
||||
machine provenance is mono; the human decision text remains sans.
|
||||
|
||||
## 10. workers
|
||||
|
||||
mockup: `final/06-workers.png`
|
||||
|
||||
purpose: answer "where can work run and is that execution machinery usable?"
|
||||
|
||||
### worker list fields
|
||||
|
||||
- worker/harness identity
|
||||
- online/idle/busy/offline/degraded
|
||||
- harness type
|
||||
- current task when leased
|
||||
- phase
|
||||
- lease or heartbeat age
|
||||
|
||||
intentionally omit generic infrastructure metrics such as disk, memory, cpu, and uptime unless orchestra directly uses them for eligibility.
|
||||
|
||||
### worker detail
|
||||
|
||||
- current session/task
|
||||
- pane id
|
||||
- lease epoch/remaining
|
||||
- last verified progress
|
||||
- worker revision
|
||||
- herdr status
|
||||
- last heartbeat
|
||||
- projects declared by the worker
|
||||
- declared capabilities if orchestra actually consumes them
|
||||
|
||||
### capabilities
|
||||
|
||||
capabilities must come from worker registration/project configuration, never be guessed from host inspection.
|
||||
|
||||
the ui labels them **declared capabilities** and can expose registration provenance.
|
||||
|
||||
### registration
|
||||
|
||||
registration is worker-driven:
|
||||
|
||||
1. worker starts with coordinator credentials/configuration
|
||||
2. worker announces identity, revision, harnesses, projects, capacity and declared capabilities
|
||||
3. coordinator records it and heartbeat updates liveness
|
||||
|
||||
`add worker` should therefore be a registration guide/bootstrap flow, not a magical browser-side creation of a remote process.
|
||||
|
||||
## 11. projects
|
||||
|
||||
mockup: `final/07-projects.png`
|
||||
|
||||
purpose: group long-running repository/workflow configuration and its active work.
|
||||
|
||||
### project list
|
||||
|
||||
- project name
|
||||
- short description
|
||||
- state
|
||||
- active/total task count
|
||||
- attention/decision count
|
||||
- last meaningful activity
|
||||
|
||||
avoid aggregate artifact counts or infrastructure metrics unless they help choose an action.
|
||||
|
||||
### project detail
|
||||
|
||||
- description
|
||||
- source/repository identity
|
||||
- configured worker/harness affinity
|
||||
- workflow phases
|
||||
- trajectory-gate policy
|
||||
- quality gate
|
||||
- human reconcile source
|
||||
- safe-operation policy
|
||||
- active tasks
|
||||
- recent meaningful events
|
||||
|
||||
configuration edits belong in the project detail, while global defaults remain in settings.
|
||||
|
||||
## 12. review
|
||||
|
||||
mockup: `final/08-review.png`
|
||||
|
||||
purpose: the operator inbox: **what needs me to look at something and judge?**
|
||||
|
||||
workers show machinery. review shows human attention.
|
||||
|
||||
### review item types
|
||||
|
||||
- trajectory decision
|
||||
- human blocker / decision request
|
||||
- code/pr review
|
||||
- ai review requiring operator override
|
||||
- operator-required lifecycle state
|
||||
- closed pr without merge
|
||||
- information/fyi, optionally hidden by default
|
||||
|
||||
### list fields
|
||||
|
||||
- type
|
||||
- task/item
|
||||
- why it is here
|
||||
- project
|
||||
- priority
|
||||
- age
|
||||
|
||||
worker/pane is secondary metadata, not a primary column.
|
||||
|
||||
### detail drawer/page
|
||||
|
||||
show the minimum evidence needed to act:
|
||||
|
||||
- why this needs the operator
|
||||
- effective task goal/constraints relevant to the decision
|
||||
- proposed options when applicable
|
||||
- relevant plan/diff/review evidence
|
||||
- provenance and age
|
||||
- comments/context drill-down
|
||||
|
||||
primary action names must match the lifecycle action: make decision, answer blocker, review pr, request changes, retry, etc.
|
||||
|
||||
snooze/defer may exist for attention management but must not silently mutate underlying task truth.
|
||||
|
||||
## 13. settings
|
||||
|
||||
mockup: `final/09-settings.png`
|
||||
|
||||
purpose: configure orchestra itself, project-independent defaults, integrations, policies, and operator/security settings.
|
||||
|
||||
### tabs
|
||||
|
||||
1. general
|
||||
2. appearance
|
||||
3. notifications
|
||||
4. access & security
|
||||
5. integrations
|
||||
6. agents & tools
|
||||
7. policies
|
||||
8. advanced
|
||||
|
||||
### general
|
||||
|
||||
- instance name/environment
|
||||
- locale/time display
|
||||
- default task preferences
|
||||
- default workflow policy
|
||||
- event/artifact retention
|
||||
- storage/backup state only if orchestra owns it
|
||||
|
||||
### appearance
|
||||
|
||||
- theme: system/dark/light
|
||||
- ethos remains fixed; no arbitrary shell redesign controls
|
||||
- accessibility and reduced motion
|
||||
|
||||
### access & security
|
||||
|
||||
- surface tokens/status
|
||||
- trusted review actors
|
||||
- ignored actors
|
||||
- operator sessions
|
||||
- agent-surface auth
|
||||
- secret/config source status without exposing secret values
|
||||
|
||||
### integrations
|
||||
|
||||
- gitea
|
||||
- vikunja
|
||||
- notification sinks
|
||||
- source reconciliation state
|
||||
|
||||
show endpoint/identity and last successful interaction in mono.
|
||||
|
||||
### agents & tools
|
||||
|
||||
- harness definitions
|
||||
- adapter/runtime configuration
|
||||
- tmux/herdr ownership
|
||||
- default launch behavior
|
||||
- context occupancy thresholds
|
||||
|
||||
### policies
|
||||
|
||||
- work phase defaults
|
||||
- trajectory gate
|
||||
- bounded decision request budget
|
||||
- retry policy
|
||||
- reconcile failure handoff threshold
|
||||
- quality/review/submission policy
|
||||
|
||||
### advanced
|
||||
|
||||
- event-store diagnostics
|
||||
- snapshots/replay
|
||||
- raw configuration
|
||||
- migration/status information
|
||||
- destructive operator actions in a visually separated danger zone
|
||||
|
||||
## 14. global command surface
|
||||
|
||||
`cmd+k` is the shared ethos command surface and should handle navigation plus safe operator actions.
|
||||
|
||||
examples:
|
||||
|
||||
- open task by id/title
|
||||
- open worker/pane
|
||||
- show decisions for task
|
||||
- steer/correct active task
|
||||
- answer current blocker
|
||||
- retry failed task
|
||||
- open review item
|
||||
|
||||
high-risk lifecycle actions require explicit confirmation and should not be the first fuzzy-search result for ordinary text.
|
||||
|
||||
## 15. status grammar
|
||||
|
||||
status is compact and consistent across screens.
|
||||
|
||||
recommended semantic colors:
|
||||
|
||||
- green: verified healthy/completed/accepted
|
||||
- violet: current/selected/orchestra-controlled active state
|
||||
- amber: waiting/degraded/operator attention
|
||||
- red: failed/refused/danger
|
||||
- gray: idle/unknown/historical
|
||||
|
||||
color never carries the entire meaning; every state also has text/iconography.
|
||||
|
||||
## 16. evidence drill-down
|
||||
|
||||
any important derived claim should be traceable without ssh.
|
||||
|
||||
from relevant detail screens the operator should be able to reach:
|
||||
|
||||
- event(s) that established the state
|
||||
- exact generated launch context
|
||||
- sealed research/plan/review/submission artifact refs
|
||||
- git sha/ref
|
||||
- worker journal observations
|
||||
- input confirmation receipt
|
||||
- pane capture/terminal
|
||||
- source comment/review provenance
|
||||
|
||||
raw evidence is accessible but not visible by default.
|
||||
|
||||
## 17. copy rules
|
||||
|
||||
- labels describe what the operator controls, not implementation class names
|
||||
- machine values remain exact and honest
|
||||
- errors state what failed and what consequence follows
|
||||
- no "something went wrong"
|
||||
- no spinner where a queue depth, heartbeat age, poll timestamp, retry time, or progress value exists
|
||||
|
||||
examples:
|
||||
|
||||
- `source reconcile failed · retry at 14:06:00`
|
||||
- `worker heartbeat 42s old · ineligible after 60s`
|
||||
- `phase request refused · next phase is research`
|
||||
- `pane lost at 13:42:18 · lease expires in 09:14`
|
||||
|
||||
## 18. responsive behavior
|
||||
|
||||
### tablet/mobile priorities
|
||||
|
||||
keep visible:
|
||||
|
||||
1. attention state
|
||||
2. task title/phase
|
||||
3. latest human authority
|
||||
4. live execution state
|
||||
5. primary action
|
||||
|
||||
collapse or drill down:
|
||||
|
||||
- long acceptance lists
|
||||
- complete workflow history
|
||||
- detailed lease metadata
|
||||
- worker journal
|
||||
- raw events
|
||||
- project configuration
|
||||
|
||||
terminal becomes its own full-screen route and should not be squeezed into a card.
|
||||
|
||||
## 19. implementation order
|
||||
|
||||
recommended web-ui build order:
|
||||
|
||||
1. ethos shell + orchestra tokens/motif
|
||||
2. dashboard
|
||||
3. tasks list
|
||||
4. task detail
|
||||
5. live pane
|
||||
6. review inbox
|
||||
7. decisions list/detail
|
||||
8. workers
|
||||
9. projects
|
||||
10. settings
|
||||
11. global command surface
|
||||
12. evidence/raw-event drill-down
|
||||
|
||||
build each screen against real orchestra api data rather than static ui-only state as early as practical.
|
||||
|
||||
## 20. verification
|
||||
|
||||
for every screen:
|
||||
|
||||
- render and visually inspect pixels
|
||||
- check desktop and <=640px mobile layout
|
||||
- assert no horizontal overflow
|
||||
- verify mono/sans split visibly
|
||||
- verify accent is signal-only
|
||||
- verify no blur/backdrop-filter exists
|
||||
- verify empty/error/loading states show real machine state
|
||||
- compare implemented screen against the corresponding mockup as direction, not pixel-perfect contract
|
||||
|
||||
mockups communicate hierarchy, density, and interaction intent. orchestra state and the ethos design laws are the actual specification.
|
||||