Record the F28 proof and the two defects setting it up exposed
F28's worker-restart direction is live-proven: three restarts, one pane pid, one lease epoch, no lifecycle events appended. The runtime-death direction is still running and will be appended when it lands. F29 and F30 are both fixed and both unbuilt. Run 5 needs a deployment carrying them, or it hits F30 the first time a handoff artifact is rejected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
This commit is contained in:
@@ -1186,3 +1186,141 @@ the 409 classified and routed to `sendPrompt`. The request file survived the
|
|||||||
failed send, which is the correct branch.
|
failed send, which is the correct branch.
|
||||||
|
|
||||||
Not proven: the receipt itself, because F28 had already destroyed the pane.
|
Not proven: the receipt itself, because F28 had already destroyed the pane.
|
||||||
|
|
||||||
|
## F28 proof and two defects it uncovered, 2026-08-28
|
||||||
|
|
||||||
|
The unit split is installed on workpc. F28's worker-restart direction is
|
||||||
|
proven live. The runtime-death direction is in progress at the time of
|
||||||
|
writing, and its result is appended below. Two new defects, F29 and F30, were found while setting the proof
|
||||||
|
up. Neither was visible in tests.
|
||||||
|
|
||||||
|
### The tmux unit, as installed
|
||||||
|
|
||||||
|
The staged `Type=forking` draft was wrong in a way the earlier entry did not
|
||||||
|
catch: without `PIDFile=`, `systemctl show orchestra-tmux -p MainPID` returned
|
||||||
|
`0`. Systemd had no main process to watch, so a crashed server would read as a
|
||||||
|
clean exit and `Restart=on-failure` would never fire.
|
||||||
|
|
||||||
|
`tmux -D` runs the server in the foreground, which gives systemd the real pid.
|
||||||
|
It also turns `exit-empty` off, so the synthetic `orchestra-runtime` session is
|
||||||
|
no longer needed to hold the server open past its last agent pane. Nothing in
|
||||||
|
the tree referenced that session.
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
User=kami
|
||||||
|
ExecStart=/usr/bin/tmux -D -L orchestra
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=1
|
||||||
|
```
|
||||||
|
|
||||||
|
Verified after install: `MainPID=3073848`, cgroup holds only the server, and
|
||||||
|
the server stays alive with zero sessions.
|
||||||
|
|
||||||
|
`deploy/orchestra-worker.service` said `User=orchestra` while the installed
|
||||||
|
unit ran as `kami`. Since the socket is `/tmp/tmux-$UID`, that mismatch would
|
||||||
|
have pointed the worker at a different server. Corrected in `fc4c29f`.
|
||||||
|
|
||||||
|
### F28 proof, live
|
||||||
|
|
||||||
|
Vehicle: disposable task `06G474FXXFW7V54RGF70ZS3SRG`, forced onto
|
||||||
|
`workpc-claude`, leased at 20:05:07Z under epoch `06G49MW2CHNVT01SHEKVYJ2170`,
|
||||||
|
pane `orchestra-06g474fxxfw7v54rgf70zs3srg-59bf1ee9:1.0`, pane pid `3012539`.
|
||||||
|
|
||||||
|
| Assertion | Result |
|
||||||
|
|---|---|
|
||||||
|
| Pane survives worker restart | 3 restarts, pid `3012539` every time |
|
||||||
|
| Same agent identity | unchanged |
|
||||||
|
| Same lease epoch continues | `06G49MW2CHNVT01SHEKVYJ2170` |
|
||||||
|
| Worker reconciles lease and pane | `leases` and `sessions` rehydrated each time |
|
||||||
|
| No `TaskReleased` from a restart | event log empty after seq 385 |
|
||||||
|
| No replacement `TaskLeased` | same |
|
||||||
|
| Subsequent renewal succeeds | 00:25:11 local, v6 → v7, `progress_sha` recorded |
|
||||||
|
|
||||||
|
The event log is the load-bearing evidence. The only `TaskReleased` on this
|
||||||
|
task is seq 382 at 20:04:06Z, from an earlier failed launch, well before the
|
||||||
|
first restart at 20:08:33Z. The restarts appended nothing at all.
|
||||||
|
|
||||||
|
Forcing a renewal by shortening the worker's cached `until` does not work. The
|
||||||
|
worker re-hydrates the authoritative deadline from the coordinator on startup,
|
||||||
|
before any renewal tick runs. The cache is not a lever.
|
||||||
|
|
||||||
|
### Forcing a task onto one harness
|
||||||
|
|
||||||
|
`config.jsonc` gave `workpc-claude` the capabilities `["code", "review"]`, and
|
||||||
|
`workpc-opencode` `["code", "review", "opencode"]`. No capability selected
|
||||||
|
claude uniquely, so an operator test could not pin placement. `claude` was
|
||||||
|
added to `workpc-claude`, mirroring the convention opencode already followed.
|
||||||
|
Router rejections at `/v1/router/health` named the gate directly, which is what
|
||||||
|
made this a two-minute diagnosis instead of an afternoon.
|
||||||
|
|
||||||
|
### F29: the browser could not act on a leased task
|
||||||
|
|
||||||
|
`validateTransition` fences `TaskReleased`, `TaskBlocked` and `TaskCompleted`
|
||||||
|
on a leased task against the live `harness_id` and `lease_epoch`. The UI action
|
||||||
|
handler sent neither, so all three returned 409 on exactly the tasks the UI
|
||||||
|
listed them as enabled for.
|
||||||
|
|
||||||
|
Found by trying to park run 4: eight block attempts, stable v8, all 409 `task
|
||||||
|
version conflict`. The workaround was to block during the queued window, where
|
||||||
|
`t.Lease` is nil and the fence does not apply.
|
||||||
|
|
||||||
|
Fixed in `efd0a5e`. The fence exists to reject a stale writer, not the
|
||||||
|
operator, so the handler now carries the lease it just read. The version CAS on
|
||||||
|
the append still rejects a racing write. A missing request body also wrote into
|
||||||
|
a nil map, which is fixed alongside.
|
||||||
|
|
||||||
|
### F30: an unpushed release pinned the worker's only slot
|
||||||
|
|
||||||
|
A release transaction that never reached `anchor_pushed` has no artifact.
|
||||||
|
`tx.Ref` is empty and no successor can pick anything up. The event handler kept
|
||||||
|
its session mapping alive anyway, so once the pane was gone the mapping was
|
||||||
|
immortal. `health()` reports `ActiveTask` straight out of `w.sessions`, so the
|
||||||
|
coordinator saw the harness as permanently busy and never leased to it again.
|
||||||
|
|
||||||
|
Live on `workpc-claude`, stuck at phase `prepared` behind this error:
|
||||||
|
|
||||||
|
```
|
||||||
|
adapter: invalid handoff answer: invalid handoff authored field: prose smuggled into list
|
||||||
|
```
|
||||||
|
|
||||||
|
It produced no log line for fifteen minutes. Freeing it needed hand surgery on
|
||||||
|
`/var/lib/orchestra/worker-state/workpc-claude.json` with the worker stopped.
|
||||||
|
|
||||||
|
Fixed in `2dc90bd`. The mapping is kept only while an anchor actually exists,
|
||||||
|
and the transaction is dropped with it, since nothing can advance it.
|
||||||
|
|
||||||
|
Worth recording separately: the rejected handoff artifact came from an agent
|
||||||
|
running the operator's direct-prose stop hook. That is a live interaction
|
||||||
|
between an operator hook and the handoff contract, not a defect in either.
|
||||||
|
|
||||||
|
### Ledger
|
||||||
|
|
||||||
|
```
|
||||||
|
F6 closed
|
||||||
|
F14 closed
|
||||||
|
F15 closed by detection, transport fix pending live proof
|
||||||
|
F16 closed, both branches live-proven
|
||||||
|
F17 fixed, isolated live proof
|
||||||
|
F18 open observability, non-blocking, cost diagnosis time twice this session
|
||||||
|
F19 fixed
|
||||||
|
F20 fixed, live receipt still pending
|
||||||
|
F21 reading and artifact halves live-proven; accepted transition pending
|
||||||
|
F22 fixed, live proof pending
|
||||||
|
F23 closed as already implemented
|
||||||
|
F24 open correctness, dormant on current topology
|
||||||
|
F25 fixed, live-proven
|
||||||
|
F26 fixed, live-proven
|
||||||
|
F27 fixed, live-proven
|
||||||
|
F28 worker-restart direction live-proven; runtime-death direction pending
|
||||||
|
F29 fixed in efd0a5e, needs a coordinator rebuild
|
||||||
|
F30 fixed in 2dc90bd, needs a worker rebuild
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deployment state
|
||||||
|
|
||||||
|
Both fixes are committed and unbuilt. The running coordinator and worker are
|
||||||
|
still `fbaaf79`. Run 5 should start from a deployment carrying `efd0a5e` and
|
||||||
|
`2dc90bd`, or it will hit F30 again the first time a handoff artifact is
|
||||||
|
rejected.
|
||||||
|
|||||||
Reference in New Issue
Block a user