kami/orchestra
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Record run 12, the expired release proven both ways

Browse Source 
F57 accepted, F58 and F59 abandoned. The rig, the two live traces, and
the correction to the operator-lifecycle entry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
This commit is contained in:
kami
2026-08-28 23:16:56 +04:00
parent 8e37989526
commit b317ecb1bd
1 changed files with 144 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BURNIN.md
+144
View File
@@ -2130,3 +2130,147 @@ and deployed, not exercised by a full run.
intervention timed against the implement phase will lose that race.
- `sudo` is unavailable in this sandbox, so every worker-side fix needs the
operator. Batch them: one install per run, not one per defect.
## Run 12, 2026-08-28: the expired release, proven both ways
Run 12 exists to settle one question the freeze left open: **a release
transaction whose lease has expired can never commit.** Run 10 ended that way
and lost a finished task. The fix and its guard were both proven live on
`test-e2e`, on a task built for the purpose.
### F57: an expired lease could never commit the anchor it had already pushed
`6565b9f`. A release pushes the anchor first and commits second. When the lease
died in between, the commit could never land, for two reasons at once:
- The worker sent the epoch from `w.leases`, which the expiry replay had
already deleted, so the request carried an empty epoch.
- The coordinator refused any `/handoff` without a live owned lease, and expiry
had already moved the version the worker held.
The epoch now belongs to the release transaction, so it survives the lease.
`TaskReleased` retains the ending epoch as `Task.LastLeaseEpoch`, and
`lateHandoffAccepted` lets exactly that owner commit while the task is queued,
unleased, and carrying no handoff of its own.
### The rig, and why suspending the worker cannot produce this
The ordering only exists inside one call: transaction opened, anchor pushing,
commit not yet sent. The event replay runs at the top of every tick, so any
expiry the worker learns about before pushing discards the transaction and
quarantines the session, by the F30 rule. A `SIGSTOP` therefore cannot make it
happen; the pause has to land inside the push.
What worked: poll `/var/lib/orchestra/worker-state/workpc-claude.json` at 2ms
and fire `POST /v1/tasks/<id>/release` the instant a transaction appears at
`prepared`. The forced expiry is the real expiry path, not a shortcut.
`validateTransition` accepts `reason: lease_expired` only when the payload
binds the current `harness_id` and `lease_epoch`, which is the same rule the
coordinator's own sweep obeys.
Task `06G4KENHXY12M5BNC5TXAF3MXR`, transaction `06G4KFK9EV4JZEPFEVEB65K2KM`:
```text
19:00:10.742 transaction opens, lease_epoch 06G4KF6HC66AGNG5HZZXCHRYPG, phase prepared
19:00:10.727 TaskReleased v15 reason=lease_expired epoch=06G4KF6HC66AGNG5HZZXCHRYPG surface=tui
19:00:11.662 TaskReleased v16 tx=06G4KFK9EV4JZEPFEVEB65K2KM hr=d4b203e9e5c0 surface=system
19:00:11.665 TaskLeased v17 epoch=06G4KFKD27HCR7MBS37M0T48AR hr=d4b203e9e5c0
19:00:15.524 TaskPickupValidated v18
```
The commit landed 935ms after the lease died. On `44ff35a`, v16 is
`409 lease not owned` and the anchor is stranded.
### The race guard, proven by making a successor win
Same rig, one step added: lease the task to `race-guard-probe` immediately
after the forced expiry, before the push finishes.
```text
19:01:30.539 transaction 06G4KFX15CW9S3967FV7R09GSM opens, epoch 06G4KFKD27HCR7MBS37M0T48AR
19:01:30.525 TaskReleased v22 reason=lease_expired
19:01:30.538 TaskLeased v23 harness=race-guard-probe epoch=06G4KFX15881BB6BX5X0YMV988
late commit -> 409 lease not owned, no v24 handoff, successor lease intact
```
A named probe harness is the cheap way to own a lease without starting an
agent. Nothing picks it up and it expires on the normal TTL.
### F58: a superseded release transaction retried forever
`03663f4`, and it is F57's own residue. That 409 is correct and permanent: the
late-handoff path fences on the epoch that expired, and the owner has moved on
twice. The worker kept asking every five seconds anyway, holding the pane and
pinning both `ActiveTask` and the single `last_error` slot. Run 10's task did
that for seven hours, which is also what hid run 11's failures.
`TaskLeased` now abandons a release transaction whose id the lease does not
carry, and quarantines its session. A successor pickup carries the
predecessor's own transaction id, so the recoverable predecessor F30 protects
is left alone.
### F59: a failed task kept its release transaction too
`8e37989`. F58 fires on `TaskLeased`, and a failed task is never leased again.
Run 12's own rig task proved the gap within ten minutes: two forced expiries
plus the probe's expiry pushed it to `retry_limit`, and it failed still holding
a transaction whose commit is refused permanently. `TaskFailed` now drops the
transaction and quarantines the session even when the anchor was pushed.
Blocked keeps the old rule, because a reopen still produces a successor that
can pick the anchor up.
A blocked task that is never reopened is therefore still able to loop. That is
run 10's case, and it is the one shape left that needs a hand.
### Corrected from the freeze
**The operator lifecycle actions do not lose a version race.** `block`,
`release` and `attention` are refused on a leased task by
`internal/store/store.go:885-901` when the payload omits `harness_id` and
`lease_epoch`. Ten attempts in 550ms all failed that way. Sending the two
fencing fields makes them succeed on the first try. The previous entry blamed a
read-then-append race, and that was wrong.
### Deployed state
| Half | Revision |
|---|---|
| Coordinator, homesrv container | `8e37989` |
| Worker, workpc systemd | `8e37989` |
```text
commit 8e37989526d8ea14088872138e32438b0df061c3
coordinator sha256 e2b3a2bb5ec374b7eae46586476712a16b43bee11039a910997d31f56f72903c
worker sha256 204f3c82b0b20aa2f86dfec3d117f4765678a86ce76bbfe43f9c80bfbd4cc245
```
The coordinator sha256 is `/app/orchestra` inside the container. Docker
compiles its own binary, so it never matches `build/orchestra`. F57 shipped on
`6565b9f` and F58 on `03663f4`; both were deployed and verified in turn, and
`8e37989` is the pair that is live.
The operator installed the `/etc/sudoers.d` line the freeze offered, so worker
installs no longer need a human. The batching advice still stands: each install
is still a restart of live state.
### Cleanup done, and one left
Two dead transactions were dropped from the worker state by hand, each with the
worker suspended and the file backed up:
| Task | Backup | Why the code could not clear it |
|---|---|---|
| `06G4GBSQ2WRGD5HGYPYZZ4TYH0` | `.bak-preclean-03663f4` | Run 10's. Blocked, so `TaskLeased` never comes. |
| `06G4KENHXY12M5BNC5TXAF3MXR` | `.bak-preclean-8e37989` | Run 12's rig task. Its `TaskFailed` was already behind the worker's cursor when F59 deployed. |
Both panes are still orphaned and need an operator kill. The worker no longer
holds either session, so nothing will close them:
```text
orchestra-06g4gbsq2wrgd5hgypyzz4tyh0-ec8111ad:1.0
orchestra-06g4kenhxy12m5bnc5txaf3mxr-*:1.0
tmux -L orchestra kill-session -t <session>
```
Task `06G4JX6MSQEP7N0D5JWW9EP5X4` is the other run 12 task and is `in_review` on
a pull request. It is real work and should be reviewed or failed, not cleaned.