From b317ecb1bd282fe159279ed2f3811bf3bf47a2ec Mon Sep 17 00:00:00 2001 From: kami Date: Fri, 28 Aug 2026 23:16:56 +0400 Subject: [PATCH] Record run 12, the expired release proven both ways F57 accepted, F58 and F59 abandoned. The rig, the two live traces, and the correction to the operator-lifecycle entry. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1 --- BURNIN.md | 144 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) diff --git a/BURNIN.md b/BURNIN.md index 9974b11..7890df2 100644 --- a/BURNIN.md +++ b/BURNIN.md @@ -2130,3 +2130,147 @@ and deployed, not exercised by a full run. intervention timed against the implement phase will lose that race. - `sudo` is unavailable in this sandbox, so every worker-side fix needs the operator. Batch them: one install per run, not one per defect. + +## Run 12, 2026-08-28: the expired release, proven both ways + +Run 12 exists to settle one question the freeze left open: **a release +transaction whose lease has expired can never commit.** Run 10 ended that way +and lost a finished task. The fix and its guard were both proven live on +`test-e2e`, on a task built for the purpose. + +### F57: an expired lease could never commit the anchor it had already pushed + +`6565b9f`. A release pushes the anchor first and commits second. When the lease +died in between, the commit could never land, for two reasons at once: + +- The worker sent the epoch from `w.leases`, which the expiry replay had + already deleted, so the request carried an empty epoch. +- The coordinator refused any `/handoff` without a live owned lease, and expiry + had already moved the version the worker held. + +The epoch now belongs to the release transaction, so it survives the lease. +`TaskReleased` retains the ending epoch as `Task.LastLeaseEpoch`, and +`lateHandoffAccepted` lets exactly that owner commit while the task is queued, +unleased, and carrying no handoff of its own. + +### The rig, and why suspending the worker cannot produce this + +The ordering only exists inside one call: transaction opened, anchor pushing, +commit not yet sent. The event replay runs at the top of every tick, so any +expiry the worker learns about before pushing discards the transaction and +quarantines the session, by the F30 rule. A `SIGSTOP` therefore cannot make it +happen; the pause has to land inside the push. + +What worked: poll `/var/lib/orchestra/worker-state/workpc-claude.json` at 2ms +and fire `POST /v1/tasks//release` the instant a transaction appears at +`prepared`. The forced expiry is the real expiry path, not a shortcut. +`validateTransition` accepts `reason: lease_expired` only when the payload +binds the current `harness_id` and `lease_epoch`, which is the same rule the +coordinator's own sweep obeys. + +Task `06G4KENHXY12M5BNC5TXAF3MXR`, transaction `06G4KFK9EV4JZEPFEVEB65K2KM`: + +```text +19:00:10.742 transaction opens, lease_epoch 06G4KF6HC66AGNG5HZZXCHRYPG, phase prepared +19:00:10.727 TaskReleased v15 reason=lease_expired epoch=06G4KF6HC66AGNG5HZZXCHRYPG surface=tui +19:00:11.662 TaskReleased v16 tx=06G4KFK9EV4JZEPFEVEB65K2KM hr=d4b203e9e5c0 surface=system +19:00:11.665 TaskLeased v17 epoch=06G4KFKD27HCR7MBS37M0T48AR hr=d4b203e9e5c0 +19:00:15.524 TaskPickupValidated v18 +``` + +The commit landed 935ms after the lease died. On `44ff35a`, v16 is +`409 lease not owned` and the anchor is stranded. + +### The race guard, proven by making a successor win + +Same rig, one step added: lease the task to `race-guard-probe` immediately +after the forced expiry, before the push finishes. + +```text +19:01:30.539 transaction 06G4KFX15CW9S3967FV7R09GSM opens, epoch 06G4KFKD27HCR7MBS37M0T48AR +19:01:30.525 TaskReleased v22 reason=lease_expired +19:01:30.538 TaskLeased v23 harness=race-guard-probe epoch=06G4KFX15881BB6BX5X0YMV988 + late commit -> 409 lease not owned, no v24 handoff, successor lease intact +``` + +A named probe harness is the cheap way to own a lease without starting an +agent. Nothing picks it up and it expires on the normal TTL. + +### F58: a superseded release transaction retried forever + +`03663f4`, and it is F57's own residue. That 409 is correct and permanent: the +late-handoff path fences on the epoch that expired, and the owner has moved on +twice. The worker kept asking every five seconds anyway, holding the pane and +pinning both `ActiveTask` and the single `last_error` slot. Run 10's task did +that for seven hours, which is also what hid run 11's failures. + +`TaskLeased` now abandons a release transaction whose id the lease does not +carry, and quarantines its session. A successor pickup carries the +predecessor's own transaction id, so the recoverable predecessor F30 protects +is left alone. + +### F59: a failed task kept its release transaction too + +`8e37989`. F58 fires on `TaskLeased`, and a failed task is never leased again. +Run 12's own rig task proved the gap within ten minutes: two forced expiries +plus the probe's expiry pushed it to `retry_limit`, and it failed still holding +a transaction whose commit is refused permanently. `TaskFailed` now drops the +transaction and quarantines the session even when the anchor was pushed. +Blocked keeps the old rule, because a reopen still produces a successor that +can pick the anchor up. + +A blocked task that is never reopened is therefore still able to loop. That is +run 10's case, and it is the one shape left that needs a hand. + +### Corrected from the freeze + +**The operator lifecycle actions do not lose a version race.** `block`, +`release` and `attention` are refused on a leased task by +`internal/store/store.go:885-901` when the payload omits `harness_id` and +`lease_epoch`. Ten attempts in 550ms all failed that way. Sending the two +fencing fields makes them succeed on the first try. The previous entry blamed a +read-then-append race, and that was wrong. + +### Deployed state + +| Half | Revision | +|---|---| +| Coordinator, homesrv container | `8e37989` | +| Worker, workpc systemd | `8e37989` | + +```text +commit 8e37989526d8ea14088872138e32438b0df061c3 +coordinator sha256 e2b3a2bb5ec374b7eae46586476712a16b43bee11039a910997d31f56f72903c +worker sha256 204f3c82b0b20aa2f86dfec3d117f4765678a86ce76bbfe43f9c80bfbd4cc245 +``` + +The coordinator sha256 is `/app/orchestra` inside the container. Docker +compiles its own binary, so it never matches `build/orchestra`. F57 shipped on +`6565b9f` and F58 on `03663f4`; both were deployed and verified in turn, and +`8e37989` is the pair that is live. + +The operator installed the `/etc/sudoers.d` line the freeze offered, so worker +installs no longer need a human. The batching advice still stands: each install +is still a restart of live state. + +### Cleanup done, and one left + +Two dead transactions were dropped from the worker state by hand, each with the +worker suspended and the file backed up: + +| Task | Backup | Why the code could not clear it | +|---|---|---| +| `06G4GBSQ2WRGD5HGYPYZZ4TYH0` | `.bak-preclean-03663f4` | Run 10's. Blocked, so `TaskLeased` never comes. | +| `06G4KENHXY12M5BNC5TXAF3MXR` | `.bak-preclean-8e37989` | Run 12's rig task. Its `TaskFailed` was already behind the worker's cursor when F59 deployed. | + +Both panes are still orphaned and need an operator kill. The worker no longer +holds either session, so nothing will close them: + +```text +orchestra-06g4gbsq2wrgd5hgypyzz4tyh0-ec8111ad:1.0 +orchestra-06g4kenhxy12m5bnc5txaf3mxr-*:1.0 +tmux -L orchestra kill-session -t +``` + +Task `06G4JX6MSQEP7N0D5JWW9EP5X4` is the other run 12 task and is `in_review` on +a pull request. It is real work and should be reviewed or failed, not cleaned.