Own the tmux execution runtime as its own service
F28. The worker spawns the tmux server on its first command, so the server and every agent pane sit in the worker unit's cgroup. Restarting the worker destroyed the sessions it was restarting to manage, and F16's missing-pane branch has been firing on deployment rather than on real execution loss. KillMode is not the fix. Under mixed systemd still SIGKILLs the cgroup remainder once the main process exits, and process only encodes accidental orphaning. The runtime becomes its own service instead. The worker gains After= and Wants= on it, ordering only: a worker that finds the runtime missing must report that rather than be stopped by it. The unit holds an idle session so the server outlives its last agent pane. User must match between the units, since the socket lives under /tmp/tmux-$UID. The installed worker on workpc runs as kami while this file still says orchestra; the staged copy is set to kami to match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
This commit is contained in:
@@ -1093,3 +1093,96 @@ F25 fixed, tests only
|
||||
F26 fixed, tests only
|
||||
F27 fixed, tests only
|
||||
```
|
||||
|
||||
### F28, deployment: the execution runtime shares the worker's cgroup
|
||||
|
||||
Worker restart destroys agent sessions because the tmux execution runtime
|
||||
shares the worker service cgroup. tmux must be independently lifecycle-managed.
|
||||
|
||||
The worker spawns the server implicitly on its first tmux command, so the
|
||||
server and every pane land in `orchestra-worker.service`:
|
||||
|
||||
```text
|
||||
orchestra-worker.service
|
||||
├── orchestra-worker
|
||||
└── tmux server
|
||||
└── claude panes
|
||||
```
|
||||
|
||||
`KillMode` does not fix this. Under `mixed` systemd still sends the final
|
||||
SIGKILL to whatever remains in the cgroup once the main process exits, and
|
||||
`process` only encodes accidental orphaning, which systemd itself discourages.
|
||||
An earlier draft of this entry proposed `mixed` and was wrong.
|
||||
|
||||
The split:
|
||||
|
||||
```text
|
||||
orchestra-worker.service
|
||||
└── orchestra-worker
|
||||
|
||||
orchestra-tmux.service
|
||||
└── tmux -L orchestra
|
||||
└── claude sessions
|
||||
```
|
||||
|
||||
`deploy/orchestra-tmux.service` is new. `deploy/orchestra-worker.service` gains
|
||||
`After=`/`Wants=` on it, ordering only: a worker that finds the runtime missing
|
||||
must report that rather than be stopped by it. The unit carries an idle
|
||||
`orchestra-runtime` session so the server outlives its last agent pane.
|
||||
|
||||
Probed live on a throwaway socket: `tmux -L <s> new-session -d` leaves the
|
||||
server running under its own pid after the client exits, so `Type=forking`
|
||||
resolves a main pid. `systemd-analyze verify` passes.
|
||||
|
||||
`User` must match between the two units, because the socket lives under
|
||||
`/tmp/tmux-$UID`. The installed worker unit on workpc runs as `kami` while
|
||||
`deploy/orchestra-worker.service` still says `orchestra`. The staged copy at
|
||||
`~/orchestra-deploy/orchestra-tmux.service` is set to `kami` to match reality.
|
||||
|
||||
### What F28 retroactively explains
|
||||
|
||||
- Run 3's `no server running` at 11:02:16 followed the 10:45:26 worker restart.
|
||||
- The same at 13:06:22.
|
||||
- Run 4's refusal could not be delivered at 13:32:16, seconds after the
|
||||
13:31:51 restart.
|
||||
- F16's missing-pane branch has been firing on deployment, not on real
|
||||
execution-runtime loss. Its proof stands as written, but the trigger was
|
||||
self-inflicted.
|
||||
|
||||
### F28 proof, to run after the split
|
||||
|
||||
```text
|
||||
1. start task and obtain live pane P
|
||||
2. record pane/session identity
|
||||
3. systemctl restart orchestra-worker
|
||||
4. assert pane P still exists with identical agent identity
|
||||
5. new worker starts
|
||||
6. worker reconciles lease + pane P
|
||||
7. no TaskReleased / new lease caused merely by worker restart
|
||||
8. agent continues under the same lease epoch
|
||||
```
|
||||
|
||||
Then separately, to restore the meaning of F16's missing-pane branch:
|
||||
|
||||
```text
|
||||
restart orchestra-tmux
|
||||
→ pane disappears
|
||||
→ F16 refuses renewal
|
||||
→ lease expires and requeues
|
||||
```
|
||||
|
||||
### Run 4 diagnostic continuation, partial
|
||||
|
||||
The chain ran and failed only at the last step, at 13:32:16:
|
||||
|
||||
```
|
||||
deliver phase refusal 06G46P6KE25Y04VVF7VRZMHZ78: tmux send-keys ... -l -- Orchestra refused your phase request: phase request refused: task 06G46P6KE25Y04VVF7VRZMHZ78 may only move to "research", not "implement"
|
||||
...: no server running on /tmp/tmux-1000/orchestra: exit status 1
|
||||
```
|
||||
|
||||
Live-proven by that one line: F25, the boundary ran on claude at all; F21's
|
||||
reading half; F26's premise, the coordinator naming the one valid target; F27,
|
||||
the 409 classified and routed to `sendPrompt`. The request file survived the
|
||||
failed send, which is the correct branch.
|
||||
|
||||
Not proven: the receipt itself, because F28 had already destroyed the pane.
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
# The tmux execution runtime, owned separately from the worker.
|
||||
#
|
||||
# The worker used to spawn this server implicitly on its first tmux command,
|
||||
# which put the server and every agent pane inside the worker unit's cgroup.
|
||||
# Restarting the worker then destroyed the sessions it was restarting to
|
||||
# manage: a deploy killed the run it was deploying for, and F16's missing-pane
|
||||
# branch fired on deployment rather than on real execution-runtime loss.
|
||||
#
|
||||
# KillMode does not fix that. Under `mixed` systemd still sends the final
|
||||
# SIGKILL to whatever remains in the cgroup, and `process` only encodes
|
||||
# accidental orphaning. The runtime has to be a service of its own.
|
||||
#
|
||||
# Lifecycle this gives you:
|
||||
# systemctl restart orchestra-worker -> panes survive, worker reconciles
|
||||
# systemctl stop orchestra-tmux -> panes die, deliberately
|
||||
#
|
||||
# User MUST match orchestra-worker.service. The socket lives under
|
||||
# /tmp/tmux-$UID, so a mismatch leaves the worker talking to a different
|
||||
# server, or to none.
|
||||
[Unit]
|
||||
Description=Orchestra tmux execution runtime
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=forking
|
||||
User=orchestra
|
||||
# The socket name is the worker's tmux_socket for this harness (-L, not -S).
|
||||
# Keep the two in step; the worker does not create the server any more.
|
||||
#
|
||||
# The idle session exists so the server outlives its last agent pane. Without
|
||||
# it the runtime exits whenever a task completes, and the next launch spawns a
|
||||
# fresh server back inside whichever cgroup asked for it.
|
||||
ExecStart=/usr/bin/tmux -L orchestra new-session -d -s orchestra-runtime
|
||||
ExecStop=/usr/bin/tmux -L orchestra kill-server
|
||||
# The runtime coming back empty is honest: the panes are gone either way, and
|
||||
# F16 observes that as real execution loss. Refusing to restart would instead
|
||||
# block every later launch until an operator noticed.
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -2,6 +2,11 @@
|
||||
Description=Orchestra federation worker
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Ordering only, deliberately not Requires. The execution runtime owns the
|
||||
# agent panes and must outlive a worker restart, and a worker that finds it
|
||||
# missing has to report that rather than be stopped by it.
|
||||
After=orchestra-tmux.service
|
||||
Wants=orchestra-tmux.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
|
||||
Reference in New Issue
Block a user