Record the F28 proof and the two defects setting it up exposed
F28's worker-restart direction is live-proven: three restarts, one pane pid, one lease epoch, no lifecycle events appended. The runtime-death direction is still running and will be appended when it lands. F29 and F30 are both fixed and both unbuilt. Run 5 needs a deployment carrying them, or it hits F30 the first time a handoff artifact is rejected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
This commit is contained in:
@@ -1186,3 +1186,141 @@ the 409 classified and routed to `sendPrompt`. The request file survived the
|
||||
failed send, which is the correct branch.
|
||||
|
||||
Not proven: the receipt itself, because F28 had already destroyed the pane.
|
||||
|
||||
## F28 proof and two defects it uncovered, 2026-08-28
|
||||
|
||||
The unit split is installed on workpc. F28's worker-restart direction is
|
||||
proven live. The runtime-death direction is in progress at the time of
|
||||
writing, and its result is appended below. Two new defects, F29 and F30, were found while setting the proof
|
||||
up. Neither was visible in tests.
|
||||
|
||||
### The tmux unit, as installed
|
||||
|
||||
The staged `Type=forking` draft was wrong in a way the earlier entry did not
|
||||
catch: without `PIDFile=`, `systemctl show orchestra-tmux -p MainPID` returned
|
||||
`0`. Systemd had no main process to watch, so a crashed server would read as a
|
||||
clean exit and `Restart=on-failure` would never fire.
|
||||
|
||||
`tmux -D` runs the server in the foreground, which gives systemd the real pid.
|
||||
It also turns `exit-empty` off, so the synthetic `orchestra-runtime` session is
|
||||
no longer needed to hold the server open past its last agent pane. Nothing in
|
||||
the tree referenced that session.
|
||||
|
||||
```ini
|
||||
[Service]
|
||||
Type=simple
|
||||
User=kami
|
||||
ExecStart=/usr/bin/tmux -D -L orchestra
|
||||
Restart=on-failure
|
||||
RestartSec=1
|
||||
```
|
||||
|
||||
Verified after install: `MainPID=3073848`, cgroup holds only the server, and
|
||||
the server stays alive with zero sessions.
|
||||
|
||||
`deploy/orchestra-worker.service` said `User=orchestra` while the installed
|
||||
unit ran as `kami`. Since the socket is `/tmp/tmux-$UID`, that mismatch would
|
||||
have pointed the worker at a different server. Corrected in `fc4c29f`.
|
||||
|
||||
### F28 proof, live
|
||||
|
||||
Vehicle: disposable task `06G474FXXFW7V54RGF70ZS3SRG`, forced onto
|
||||
`workpc-claude`, leased at 20:05:07Z under epoch `06G49MW2CHNVT01SHEKVYJ2170`,
|
||||
pane `orchestra-06g474fxxfw7v54rgf70zs3srg-59bf1ee9:1.0`, pane pid `3012539`.
|
||||
|
||||
| Assertion | Result |
|
||||
|---|---|
|
||||
| Pane survives worker restart | 3 restarts, pid `3012539` every time |
|
||||
| Same agent identity | unchanged |
|
||||
| Same lease epoch continues | `06G49MW2CHNVT01SHEKVYJ2170` |
|
||||
| Worker reconciles lease and pane | `leases` and `sessions` rehydrated each time |
|
||||
| No `TaskReleased` from a restart | event log empty after seq 385 |
|
||||
| No replacement `TaskLeased` | same |
|
||||
| Subsequent renewal succeeds | 00:25:11 local, v6 → v7, `progress_sha` recorded |
|
||||
|
||||
The event log is the load-bearing evidence. The only `TaskReleased` on this
|
||||
task is seq 382 at 20:04:06Z, from an earlier failed launch, well before the
|
||||
first restart at 20:08:33Z. The restarts appended nothing at all.
|
||||
|
||||
Forcing a renewal by shortening the worker's cached `until` does not work. The
|
||||
worker re-hydrates the authoritative deadline from the coordinator on startup,
|
||||
before any renewal tick runs. The cache is not a lever.
|
||||
|
||||
### Forcing a task onto one harness
|
||||
|
||||
`config.jsonc` gave `workpc-claude` the capabilities `["code", "review"]`, and
|
||||
`workpc-opencode` `["code", "review", "opencode"]`. No capability selected
|
||||
claude uniquely, so an operator test could not pin placement. `claude` was
|
||||
added to `workpc-claude`, mirroring the convention opencode already followed.
|
||||
Router rejections at `/v1/router/health` named the gate directly, which is what
|
||||
made this a two-minute diagnosis instead of an afternoon.
|
||||
|
||||
### F29: the browser could not act on a leased task
|
||||
|
||||
`validateTransition` fences `TaskReleased`, `TaskBlocked` and `TaskCompleted`
|
||||
on a leased task against the live `harness_id` and `lease_epoch`. The UI action
|
||||
handler sent neither, so all three returned 409 on exactly the tasks the UI
|
||||
listed them as enabled for.
|
||||
|
||||
Found by trying to park run 4: eight block attempts, stable v8, all 409 `task
|
||||
version conflict`. The workaround was to block during the queued window, where
|
||||
`t.Lease` is nil and the fence does not apply.
|
||||
|
||||
Fixed in `efd0a5e`. The fence exists to reject a stale writer, not the
|
||||
operator, so the handler now carries the lease it just read. The version CAS on
|
||||
the append still rejects a racing write. A missing request body also wrote into
|
||||
a nil map, which is fixed alongside.
|
||||
|
||||
### F30: an unpushed release pinned the worker's only slot
|
||||
|
||||
A release transaction that never reached `anchor_pushed` has no artifact.
|
||||
`tx.Ref` is empty and no successor can pick anything up. The event handler kept
|
||||
its session mapping alive anyway, so once the pane was gone the mapping was
|
||||
immortal. `health()` reports `ActiveTask` straight out of `w.sessions`, so the
|
||||
coordinator saw the harness as permanently busy and never leased to it again.
|
||||
|
||||
Live on `workpc-claude`, stuck at phase `prepared` behind this error:
|
||||
|
||||
```
|
||||
adapter: invalid handoff answer: invalid handoff authored field: prose smuggled into list
|
||||
```
|
||||
|
||||
It produced no log line for fifteen minutes. Freeing it needed hand surgery on
|
||||
`/var/lib/orchestra/worker-state/workpc-claude.json` with the worker stopped.
|
||||
|
||||
Fixed in `2dc90bd`. The mapping is kept only while an anchor actually exists,
|
||||
and the transaction is dropped with it, since nothing can advance it.
|
||||
|
||||
Worth recording separately: the rejected handoff artifact came from an agent
|
||||
running the operator's direct-prose stop hook. That is a live interaction
|
||||
between an operator hook and the handoff contract, not a defect in either.
|
||||
|
||||
### Ledger
|
||||
|
||||
```
|
||||
F6 closed
|
||||
F14 closed
|
||||
F15 closed by detection, transport fix pending live proof
|
||||
F16 closed, both branches live-proven
|
||||
F17 fixed, isolated live proof
|
||||
F18 open observability, non-blocking, cost diagnosis time twice this session
|
||||
F19 fixed
|
||||
F20 fixed, live receipt still pending
|
||||
F21 reading and artifact halves live-proven; accepted transition pending
|
||||
F22 fixed, live proof pending
|
||||
F23 closed as already implemented
|
||||
F24 open correctness, dormant on current topology
|
||||
F25 fixed, live-proven
|
||||
F26 fixed, live-proven
|
||||
F27 fixed, live-proven
|
||||
F28 worker-restart direction live-proven; runtime-death direction pending
|
||||
F29 fixed in efd0a5e, needs a coordinator rebuild
|
||||
F30 fixed in 2dc90bd, needs a worker rebuild
|
||||
```
|
||||
|
||||
### Deployment state
|
||||
|
||||
Both fixes are committed and unbuilt. The running coordinator and worker are
|
||||
still `fbaaf79`. Run 5 should start from a deployment carrying `efd0a5e` and
|
||||
`2dc90bd`, or it will hit F30 again the first time a handoff artifact is
|
||||
rejected.
|
||||
|
||||
Reference in New Issue
Block a user