From c8db659b45a485fdcc8941197f12e1d21f380907 Mon Sep 17 00:00:00 2001 From: kami Date: Fri, 28 Aug 2026 00:42:38 +0400 Subject: [PATCH] Record the F28 proof and the two defects setting it up exposed F28's worker-restart direction is live-proven: three restarts, one pane pid, one lease epoch, no lifecycle events appended. The runtime-death direction is still running and will be appended when it lands. F29 and F30 are both fixed and both unbuilt. Run 5 needs a deployment carrying them, or it hits F30 the first time a handoff artifact is rejected. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu --- BURNIN.md | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) diff --git a/BURNIN.md b/BURNIN.md index 1eee101..2775a44 100644 --- a/BURNIN.md +++ b/BURNIN.md @@ -1186,3 +1186,141 @@ the 409 classified and routed to `sendPrompt`. The request file survived the failed send, which is the correct branch. Not proven: the receipt itself, because F28 had already destroyed the pane. + +## F28 proof and two defects it uncovered, 2026-08-28 + +The unit split is installed on workpc. F28's worker-restart direction is +proven live. The runtime-death direction is in progress at the time of +writing, and its result is appended below. Two new defects, F29 and F30, were found while setting the proof +up. Neither was visible in tests. + +### The tmux unit, as installed + +The staged `Type=forking` draft was wrong in a way the earlier entry did not +catch: without `PIDFile=`, `systemctl show orchestra-tmux -p MainPID` returned +`0`. Systemd had no main process to watch, so a crashed server would read as a +clean exit and `Restart=on-failure` would never fire. + +`tmux -D` runs the server in the foreground, which gives systemd the real pid. +It also turns `exit-empty` off, so the synthetic `orchestra-runtime` session is +no longer needed to hold the server open past its last agent pane. Nothing in +the tree referenced that session. + +```ini +[Service] +Type=simple +User=kami +ExecStart=/usr/bin/tmux -D -L orchestra +Restart=on-failure +RestartSec=1 +``` + +Verified after install: `MainPID=3073848`, cgroup holds only the server, and +the server stays alive with zero sessions. + +`deploy/orchestra-worker.service` said `User=orchestra` while the installed +unit ran as `kami`. Since the socket is `/tmp/tmux-$UID`, that mismatch would +have pointed the worker at a different server. Corrected in `fc4c29f`. + +### F28 proof, live + +Vehicle: disposable task `06G474FXXFW7V54RGF70ZS3SRG`, forced onto +`workpc-claude`, leased at 20:05:07Z under epoch `06G49MW2CHNVT01SHEKVYJ2170`, +pane `orchestra-06g474fxxfw7v54rgf70zs3srg-59bf1ee9:1.0`, pane pid `3012539`. + +| Assertion | Result | +|---|---| +| Pane survives worker restart | 3 restarts, pid `3012539` every time | +| Same agent identity | unchanged | +| Same lease epoch continues | `06G49MW2CHNVT01SHEKVYJ2170` | +| Worker reconciles lease and pane | `leases` and `sessions` rehydrated each time | +| No `TaskReleased` from a restart | event log empty after seq 385 | +| No replacement `TaskLeased` | same | +| Subsequent renewal succeeds | 00:25:11 local, v6 → v7, `progress_sha` recorded | + +The event log is the load-bearing evidence. The only `TaskReleased` on this +task is seq 382 at 20:04:06Z, from an earlier failed launch, well before the +first restart at 20:08:33Z. The restarts appended nothing at all. + +Forcing a renewal by shortening the worker's cached `until` does not work. The +worker re-hydrates the authoritative deadline from the coordinator on startup, +before any renewal tick runs. The cache is not a lever. + +### Forcing a task onto one harness + +`config.jsonc` gave `workpc-claude` the capabilities `["code", "review"]`, and +`workpc-opencode` `["code", "review", "opencode"]`. No capability selected +claude uniquely, so an operator test could not pin placement. `claude` was +added to `workpc-claude`, mirroring the convention opencode already followed. +Router rejections at `/v1/router/health` named the gate directly, which is what +made this a two-minute diagnosis instead of an afternoon. + +### F29: the browser could not act on a leased task + +`validateTransition` fences `TaskReleased`, `TaskBlocked` and `TaskCompleted` +on a leased task against the live `harness_id` and `lease_epoch`. The UI action +handler sent neither, so all three returned 409 on exactly the tasks the UI +listed them as enabled for. + +Found by trying to park run 4: eight block attempts, stable v8, all 409 `task +version conflict`. The workaround was to block during the queued window, where +`t.Lease` is nil and the fence does not apply. + +Fixed in `efd0a5e`. The fence exists to reject a stale writer, not the +operator, so the handler now carries the lease it just read. The version CAS on +the append still rejects a racing write. A missing request body also wrote into +a nil map, which is fixed alongside. + +### F30: an unpushed release pinned the worker's only slot + +A release transaction that never reached `anchor_pushed` has no artifact. +`tx.Ref` is empty and no successor can pick anything up. The event handler kept +its session mapping alive anyway, so once the pane was gone the mapping was +immortal. `health()` reports `ActiveTask` straight out of `w.sessions`, so the +coordinator saw the harness as permanently busy and never leased to it again. + +Live on `workpc-claude`, stuck at phase `prepared` behind this error: + +``` +adapter: invalid handoff answer: invalid handoff authored field: prose smuggled into list +``` + +It produced no log line for fifteen minutes. Freeing it needed hand surgery on +`/var/lib/orchestra/worker-state/workpc-claude.json` with the worker stopped. + +Fixed in `2dc90bd`. The mapping is kept only while an anchor actually exists, +and the transaction is dropped with it, since nothing can advance it. + +Worth recording separately: the rejected handoff artifact came from an agent +running the operator's direct-prose stop hook. That is a live interaction +between an operator hook and the handoff contract, not a defect in either. + +### Ledger + +``` +F6 closed +F14 closed +F15 closed by detection, transport fix pending live proof +F16 closed, both branches live-proven +F17 fixed, isolated live proof +F18 open observability, non-blocking, cost diagnosis time twice this session +F19 fixed +F20 fixed, live receipt still pending +F21 reading and artifact halves live-proven; accepted transition pending +F22 fixed, live proof pending +F23 closed as already implemented +F24 open correctness, dormant on current topology +F25 fixed, live-proven +F26 fixed, live-proven +F27 fixed, live-proven +F28 worker-restart direction live-proven; runtime-death direction pending +F29 fixed in efd0a5e, needs a coordinator rebuild +F30 fixed in 2dc90bd, needs a worker rebuild +``` + +### Deployment state + +Both fixes are committed and unbuilt. The running coordinator and worker are +still `fbaaf79`. Run 5 should start from a deployment carrying `efd0a5e` and +`2dc90bd`, or it will hit F30 again the first time a handoff artifact is +rejected.