3 Commits

Author SHA1 Message Date
kami 99b209ba10 Record run 18: an implement successor inherits verified phase progress
The rung the last two runs missed. A five-phase task with ten named checks
kept the implement phase open long enough to rotate inside it. The successor
picked up in implement and its launch context carried the whole sealed plan,
phase-1 and phase-2 as verified, phase-3 as the first unfinished phase, and
the current human authority.

Unasked-for bonus: the progress block renders SHA staleness itself, naming
the tree each phase was verified against and the tree it is now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 17:48:17 +04:00
kami ab3258833d Record run 17: F62 proven live, from a production trigger
The rig was not needed. An implement to review phase change asked for the
handoff through the ordinary path, and the agent ignored it, which is the
exact shape F62 was written for. Every assertion held: a causal release
rather than an idle expiry, four renewals during the bounded wait, one
resend carrying the original reason, the timeout class at 10m5s, no stale
transaction, and a successor that leased normally a minute later.

Also recorded: a five-phase plan does not lengthen the implement phase, and
the state-file lever cannot be driven with systemctl restart alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 17:38:15 +04:00
kami c587f2cc8d Bound the wait for a handoff nobody answers
F62. The rotation is agent-driven: the worker asks, and the agent must write
its handoff. When the agent never does, renewals stopped on the ordinary
progress gate, the lease expired, and the task lost an attempt with nothing on
record saying a handoff had ever been requested. Run 16 showed only "agent
status idle and pane unchanged", 34 times.

The request is now stamped, and the wait around it is bounded. While Orchestra
is explicitly waiting the lease renews, because a quiet pane is the answer the
agent was told to give. The request is re-sent once after four minutes, with
the reason it was first asked with. At ten minutes the worker nacks with
failure class handoff_unanswered, and the coordinator releases the task naming
that cause instead of letting the lease die as generic idleness.

The class is known to DebtClassForFailureClass, so a harness that ignores
handoff requests accumulates as its own debt item rather than hiding inside
lease_expired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 16:21:35 +04:00
6 changed files with 378 additions and 4 deletions
+173
View File
@@ -2587,3 +2587,176 @@ file.
The ring earned its keep again: one distinct message with a count of 34, rather
than 34 overwrites of one slot.
**Fixed, not yet proven live.** The request is now stamped
(`herdr.Session.HandoffRequestedAt`) and the wait is bounded by
`watchHandoff` in the worker:
- the lease renews while Orchestra is explicitly waiting, because a quiet pane
is the answer the agent was asked for — the renewal gate's new case, bounded
by `handoffAnswerTimeout`;
- the request is re-sent once at `handoffRetryAfter` (4 minutes), with the same
reason it was first asked with;
- at 10 minutes the worker nacks with failure class `handoff_unanswered`, and
the coordinator emits `TaskReleased reason=handoff_unanswered` rather than
letting the lease die as generic idleness.
`DebtClassForFailureClass` knows the class, so a harness that repeatedly
ignores handoff requests now accumulates in the debt ledger instead of hiding
inside `lease_expired`.
Runtime proof still owed: force a request the agent will not answer, and read
the release event rather than the worker state file.
## Run 17, 2026-08-29: F62 proven live on `c587f2c`
Task `06G4V20T528ZTER7KZBVGNAYXC`, a five-phase plan
(`3b16b77054b9`) on the deployed pair, coordinator and worker both at
`c587f2c`.
### The trigger was production, not a rig
The rig planned for this run was a state-file lever. It was never needed. The
`implement -> review` phase change asked for a handoff through the ordinary
path (`rotateForPhase`, reason `phase_changed`), and the agent ignored it
because this task's brief instructed it to from phase 4 onward. Nothing touched
tmux, no process was stopped, and the pane stayed open for the whole wait.
```text
12:48:09 implement -> review, handoff requested, reason=phase_changed
12:50:09 TaskLeaseRenewed agent idle, pane capture unchanged
12:52:15 pane input confirmed the single resend, 4m06s after the request
12:52:39 TaskLeaseRenewed
12:55:14 TaskLeaseRenewed
12:57:49 TaskLeaseRenewed
12:58:14 TaskReleased reason=handoff_unanswered failure_class=handoff_unanswered
last_error="handoff requested (phase_changed) and unanswered for 10m5s"
12:59:14 TaskLeased successor leases normally
13:02:20 TaskSubmitted
```
Every assertion the operator set for this rung held:
- the release does not look like an ordinary `lease_expired`: it carries
`reason=handoff_unanswered` and `lifecycle_phase=handoff_unanswered`;
- the idle-renewal rule did not kill the wait: four renewals on an idle pane
with no capture movement, each one the gate would have refused before this
commit;
- the resend preserved the reason the request was first made with;
- exactly one resend, not one per tick;
- the timeout produced the class at `10m5s`, one tick past the bound;
- nothing stale was left: `releases: {}`, no session, no lease, and zero tmux
panes matching the id;
- the next attempt leased 60s later and reached `TaskSubmitted`.
The debt projection classified it with no new code beyond the class mapping:
```text
v1:operational:handoff_unanswered:workpc-claude:lease operational important
"handoff requested (phase_changed) and unanswered for 10m5s"
```
**Inference, not observation.** The resent prompt's text was not captured. That
it carried the `phase_changed` wording rests on the code path and the timing,
not on a pane capture. A future rig should publish a capture across the resend.
### The implement-successor rung missed again, and why
A five-phase plan does not lengthen the implement phase. This one ran 12:45:18
to 12:48:09, about 2m50s, with verifications 20 to 40 seconds apart:
```text
12:45:47 phase-1 12:46:07 phase-2 12:46:32 phase-3
12:47:12 phase-4 12:47:52 phase-5 12:48:09 implement -> review
```
The brief's explicit one-phase-at-a-time requirement was followed, and it
still took under three minutes. Phase count is not the lever; volume of
mechanical work is.
`awaiting_manual_verification` does not hold the phase open either. All five
phases carried that status and the agent moved on regardless.
### The state-file lever cannot be driven with `restart` alone
The documented procedure is stop, edit, start. The unattended sudo rule grants
`install` and `systemctl restart orchestra-worker`, with no `stop` or `start`.
With `restart` only, the running worker's five-second tick save landed between
the edit and the restart, and the edit was gone. Widening the sudoers line is
the fix.
### One cosmetic note
`Session.HandoffRequestedAt` serializes as `0001-01-01T00:00:00Z` when unset,
because `omitempty` does not omit a zero `time.Time`. The worker parses it back
to a zero value and `watchHandoff` starts the clock, so behaviour is correct.
It is the same marshalling trap the operator console hit.
## Run 18, 2026-08-29: the implement-successor rung, proven
Task `06G4VF5HZW7Q4JBM3TTY7W1Y64` on `c587f2c`, a five-phase plan whose phases
are ten named checks rather than one refactor. Volume of mechanical work is
what keeps the implement phase open; phase count does not.
### The rotation
```text
13:43:10 implement session launched
13:43:44 phase-1 verified
13:44:04 phase-2 verified
13:44:35 handoff requested, reason=milestone, delivered by watchHandoff's resend
13:45:25 TaskLeaseRenewed
13:46:51 TaskReleased handoff_ref 0b4fae6f705a anchor 8ea3c7d1ae8f
13:46:51 TaskLeased successor, same harness
13:46:55 TaskPickupValidated
13:47:06 TaskLaunchAcknowledged, still in implement
```
The lever was the worker state file, backdated past `handoffRetryAfter` so the
request went out on the next tick. Under `c587f2c` that lever now produces a
real prompt: the flag alone used to sit there unasked, which is how run 16
ended in a silent expiry.
### What the successor was given
From `.orchestra/launch.md`, written at 13:47:05:
```text
## Current phase
implement: Implement the accepted plan below. Verify as you go.
## Verified git state
- head: 8ea3c7d1ae8fd960897344694645117adbd8ee82
- uncommitted changes: false
## Current human decisions
None recorded. Work from the goal and acceptance above.
## Plan progress
Orchestra established this by running the plan's own verification. You cannot
write it.
- phase-1 (...): automated checks passed at d3acd4989a20, stale because the
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
- phase-2 (...): automated checks passed at d3acd4989a20, stale because the
tree is now at 8ea3c7d1ae8f, waiting for the human to confirm the manual steps
- phase-3 (Checks 7 to 8): not started
- phase-4 (...): not started
- phase-5 (...): not started
```
Everything the rung asked for is there: the complete accepted plan with all
five phases, their files, changes and verification; the phases already
verified; the first unfinished phase; and the current human authority. The
launch context also carries the accepted research and its dead ends.
**SHA staleness renders itself.** Neither phase is reported as simply passed.
Each says the checks passed at `d3acd4989a20` and are stale because the tree
has moved to `8ea3c7d1ae8f`. That is half of the manual-verification rung
observed without being asked for.
### The lever still loses a race, sometimes
`systemctl restart` alone leaves a five-second window in which the running
worker's tick save can clobber the edit. It clobbered run 17's attempt and
survived run 18's. Stop, edit, start is the reliable sequence, and it needs
`systemctl stop` and `start` in the unattended sudo rule.
+110
View File
@@ -0,0 +1,110 @@
package main
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"orchestra/internal/domain"
"orchestra/internal/federation"
"orchestra/internal/herdr"
)
// F62. Run 16: the agent was asked to hand off, never wrote HANDOFF.md,
// renewals stopped, and the lease died as ordinary idleness. Waiting is now
// bounded: re-ask once, then give the task up with a class that says why.
func TestUnansweredHandoffIsRetriedThenGivenUp(t *testing.T) {
var nack map[string]any
w, backend, _, done := phaseWorker(t, func(rw http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/nack") {
_ = json.NewDecoder(r.Body).Decode(&nack)
}
rw.Write([]byte(`{}`))
})
defer done()
ctx := context.Background()
requested := func(ago time.Duration) herdr.Session {
s := w.sessions["task"]
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
s.HandoffRequestedAt = time.Now().UTC().Add(-ago)
w.sessions["task"] = s
return s
}
// Still inside the answering window: nothing said, nothing given up.
if s, gaveUp := w.watchHandoff(ctx, "task", requested(time.Minute)); gaveUp || s.HandoffRetried {
t.Fatalf("gave up while still waiting: gaveUp=%v session=%+v", gaveUp, s)
}
if len(backend.prompts) != 0 {
t.Fatalf("re-asked too early: %q", backend.prompts)
}
// Past the retry point: asked again, exactly once.
s, gaveUp := w.watchHandoff(ctx, "task", requested(handoffRetryAfter+time.Minute))
if gaveUp || !s.HandoffRetried || len(backend.prompts) != 1 {
t.Fatalf("retry: gaveUp=%v retried=%v prompts=%q", gaveUp, s.HandoffRetried, backend.prompts)
}
if _, gaveUp = w.watchHandoff(ctx, "task", s); gaveUp || len(backend.prompts) != 1 {
t.Fatalf("re-asked every tick: %q", backend.prompts)
}
// Past the bound: a causal reclaim, and no lease left to renew.
s = requested(handoffAnswerTimeout + time.Second)
s.HandoffRetried = true
w.sessions["task"] = s
if _, gaveUp = w.watchHandoff(ctx, "task", s); !gaveUp {
t.Fatal("an unanswered handoff waited forever")
}
if nack["failure_class"] != "handoff_unanswered" {
t.Fatalf("nack = %+v", nack)
}
if detail, _ := nack["last_error"].(string); !strings.Contains(detail, "phase_changed") {
t.Fatalf("the reclaim does not name the request: %q", detail)
}
if _, held := w.leases["task"]; held {
t.Fatal("the given-up task kept its lease")
}
}
// The lease must survive the wait it was asked to make: an idle pane is the
// answer Orchestra requested, not evidence of an agent that stopped working.
func TestWaitingForAHandoffKeepsTheLease(t *testing.T) {
renewals := 0
api := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
renewals++
rw.Write([]byte(`{}`))
}))
defer api.Close()
backend := &recordingBackend{status: "idle", progress: "same screen"}
w := &worker{
api: federation.Client{BaseURL: api.URL, WorkerID: "h", Token: "t"},
backend: backend,
harness: "claude",
sessions: map[string]herdr.Session{"task": {PaneID: "pane", HandoffRequested: true, HandoffRequestedAt: time.Now().UTC()}},
leases: map[string]lease{"task": {Epoch: "e", Version: 1, Until: time.Now(), ProgressSHA: domain.Hash([]byte("same screen"))}},
quarantined: map[string]bool{},
statePath: filepath.Join(t.TempDir(), "state.json"),
}
w.renewLeases(context.Background())
if renewals != 1 {
t.Fatalf("a lease waiting on a requested handoff renewed %d times, want 1", renewals)
}
// Past the bound the exemption stops: watchHandoff has given the task up
// by then, and nothing keeps an unanswered request alive.
s := w.sessions["task"]
s.HandoffRequestedAt = time.Now().UTC().Add(-handoffAnswerTimeout - time.Second)
w.sessions["task"] = s
l := w.leases["task"]
l.Until = time.Now()
w.leases["task"] = l
w.renewLeases(context.Background())
if renewals != 1 {
t.Fatalf("the exemption outlived its bound: renewals=%d", renewals)
}
}
+80 -3
View File
@@ -174,6 +174,7 @@ func releaseBackoff(attempts int) time.Duration {
}
return d
}
type projectConfig struct {
Repo string `json:"repo"`
Root string `json:"worktree_root"`
@@ -673,6 +674,13 @@ func (w *worker) releaseReady(ctx context.Context) {
w.advanceRelease(ctx, id, s)
continue
}
if s.HandoffRequested {
next, gaveUp := w.watchHandoff(ctx, id, s)
if gaveUp {
continue
}
s = next
}
w.rotationTick(ctx, id, s)
}
}
@@ -772,7 +780,7 @@ func (w *worker) rotationTick(ctx context.Context, id string, s herdr.Session) {
w.recordError(fmt.Errorf("rotation %s threshold prompt: %w", id, err))
return
}
s.HandoffRequested, s.HandoffReason = true, d.Reason
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, d.Reason, time.Now().UTC()
w.sessions[id] = s
_ = w.save()
}
@@ -1166,6 +1174,70 @@ func (w *worker) submit(ctx context.Context, id string, s herdr.Session, e compl
// the directory also holds for a worktree that has no inner .gitignore.
const stageExclude = ":!.orchestra"
// F62: a requested handoff nobody answers was invisible. Renewals stopped,
// the lease expired, and the task lost an attempt with nothing on record
// saying a handoff had ever been asked for — worker health showed only "agent
// status idle and pane unchanged", 34 times in run 16.
const (
handoffRetryAfter = 4 * time.Minute
handoffAnswerTimeout = 10 * time.Minute
)
// watchHandoff bounds the wait for an agent's handoff answer: re-send the
// request once, then give the task up with a class that names the cause. It
// returns the session to keep using and whether the task was given up.
func (w *worker) watchHandoff(ctx context.Context, id string, s herdr.Session) (herdr.Session, bool) {
if s.HandoffRequestedAt.IsZero() {
// A session persisted before the stamp existed, or requested by a path
// that does not set it. Start the clock now rather than time out a
// request retroactively.
s.HandoffRequestedAt = time.Now().UTC()
w.sessions[id] = s
_ = w.save()
return s, false
}
waited := time.Since(s.HandoffRequestedAt)
if waited < handoffRetryAfter {
return s, false
}
if waited < handoffAnswerTimeout {
if s.HandoffRetried || w.executionBackend() == nil {
return s, false
}
a := herdr.CLIAdapter{Backend: w.executionBackend(), Harness: w.harness}
var err error
if s.HandoffReason != "" {
err = a.RequestHandoffReason(ctx, s, s.HandoffReason, nil)
} else {
err = a.RequestHandoff(ctx, s)
}
if err != nil {
w.recordError(fmt.Errorf("handoff %s re-request: %w", id, err))
return s, false
}
s.HandoffRetried = true
w.sessions[id] = s
_ = w.save()
w.recordError(fmt.Errorf("handoff %s (%s) unanswered for %s: request re-sent", id, s.HandoffReason, waited.Round(time.Second)))
return s, false
}
l, ok := w.leases[id]
if !ok {
return s, false
}
detail := fmt.Sprintf("handoff requested (%s) and unanswered for %s", s.HandoffReason, waited.Round(time.Second))
if err := w.api.NackStart(ctx, id, l.Epoch, l.Version, "handoff_unanswered", detail, w.sessionEvidence(ctx, id, s)); err != nil {
w.recordError(fmt.Errorf("handoff timeout %s: %w", id, err))
return s, false
}
// The coordinator answers with TaskReleased; its replay quarantines the
// pane. Drop the lease here so nothing renews it in the meantime.
delete(w.leases, id)
_ = w.save()
w.recordError(errors.New(detail))
return s, true
}
func (w *worker) renewLeases(ctx context.Context) {
if w.executionBackend() == nil {
return
@@ -1199,6 +1271,11 @@ func (w *worker) renewLeases(ctx context.Context) {
case l.ProgressSHA == "":
// First renewal has no baseline to compare against. Record one and
// allow this renewal; the next one must show real movement.
case s.HandoffRequested && !s.HandoffRequestedAt.IsZero() && time.Since(s.HandoffRequestedAt) < handoffAnswerTimeout:
// Orchestra told this agent to stop and write its handoff. A quiet
// pane is the answer it was asked for, so the lease is held while
// the wait is explicitly bounded (F62). Past the bound the case
// stops matching and watchHandoff has already given the task up.
default:
w.recordError(fmt.Errorf("lease %s not renewed: agent status %s and pane unchanged since the last renewal", taskID, status))
continue
@@ -1932,7 +2009,7 @@ func (w *worker) federatedTurn(ctx context.Context, id string, a herdr.Adapter,
w.recordError(fmt.Errorf("reconcile failure handoff %s: %w", id, err))
return
}
s.HandoffRequested, s.HandoffReason = true, "reconcile_failure"
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "reconcile_failure", time.Now().UTC()
w.sessions[id] = s
_ = w.save()
return
@@ -2009,7 +2086,7 @@ func (w *worker) rotateForPhase(ctx context.Context, id string, a herdr.Adapter,
w.recordError(fmt.Errorf("phase rotation %s: %w", id, err))
return
}
s.HandoffRequested, s.HandoffReason = true, "phase_changed"
s.HandoffRequested, s.HandoffReason, s.HandoffRequestedAt = true, "phase_changed", time.Now().UTC()
w.sessions[id] = s
_ = w.save()
log.Printf("phase changed for %s: session rotating", id)
+6
View File
@@ -1589,6 +1589,12 @@ func main() {
case "invalid_handoff":
typ = "TaskBlocked"
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonHandoffValidation), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_nacked", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
case "handoff_unanswered":
// F62: not a launch failure. The agent was asked to hand off
// and never did, so the reclaim says exactly that instead of
// arriving as an ordinary idle expiry.
typ = "TaskReleased"
p, _ = json.Marshal(map[string]any{"reason": "handoff_unanswered", "failure_class": b.FailureClass, "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "handoff_unanswered", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
case "launch_uncertain":
typ = "TaskNeedsAttention"
p, _ = json.Marshal(map[string]any{"blocker": b.LastError, "block_reason": string(domain.BlockReasonLeaseFailure), "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "lifecycle_phase": "launch_uncertain", "last_error": b.LastError, "session_evidence": b.SessionEvidence})
+1 -1
View File
@@ -251,7 +251,7 @@ func DebtClassForBlockReason(r BlockReason) (DebtClass, bool) {
// classes a worker actually emits are listed; an unknown one is not guessed at.
func DebtClassForFailureClass(f string) (DebtClass, bool) {
switch f {
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired":
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired", "handoff_unanswered":
return DebtOperational, true
case "invalid_handoff":
return DebtCorrectness, true
+8
View File
@@ -174,6 +174,14 @@ type Session struct {
// its §6.1 handoff (HandoffFile) — avoids re-sending the same prompt
// every tick while Release keeps waiting for the file to appear.
HandoffRequested bool `json:"handoff_requested,omitempty"`
// HandoffRequestedAt stamps that prompt. A request nobody answers used to
// end as an ordinary idle expiry, indistinguishable from an agent that
// never started (F62); the stamp is what makes the wait bounded and the
// giving-up causal.
HandoffRequestedAt time.Time `json:"handoff_requested_at,omitempty"`
// HandoffRetried records that the request was re-sent once, so a session
// waiting on an answer is not re-prompted every tick.
HandoffRetried bool `json:"handoff_retried,omitempty"`
// HandoffReason is selected by the coordinator when it asks for the
// semantic report. The checkout worker, rather than the harness, copies
// it into the canonical handoff it seals at release time.