Own the tmux execution runtime as its own service

F28. The worker spawns the tmux server on its first command, so the server and
every agent pane sit in the worker unit's cgroup. Restarting the worker
destroyed the sessions it was restarting to manage, and F16's missing-pane
branch has been firing on deployment rather than on real execution loss.

KillMode is not the fix. Under mixed systemd still SIGKILLs the cgroup
remainder once the main process exits, and process only encodes accidental
orphaning. The runtime becomes its own service instead.

The worker gains After= and Wants= on it, ordering only: a worker that finds
the runtime missing must report that rather than be stopped by it. The unit
holds an idle session so the server outlives its last agent pane.

User must match between the units, since the socket lives under /tmp/tmux-$UID.
The installed worker on workpc runs as kami while this file still says
orchestra; the staged copy is set to kami to match.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xsXyr5J1RACo71YeKG3Pu
This commit is contained in:
2026-08-27 17:48:28 +04:00
parent fbaaf79bb1
commit daa5d20d9b
3 changed files with 140 additions and 0 deletions
+93
View File
@@ -1093,3 +1093,96 @@ F25 fixed, tests only
F26 fixed, tests only
F27 fixed, tests only
```
### F28, deployment: the execution runtime shares the worker's cgroup
Worker restart destroys agent sessions because the tmux execution runtime
shares the worker service cgroup. tmux must be independently lifecycle-managed.
The worker spawns the server implicitly on its first tmux command, so the
server and every pane land in `orchestra-worker.service`:
```text
orchestra-worker.service
├── orchestra-worker
└── tmux server
└── claude panes
```
`KillMode` does not fix this. Under `mixed` systemd still sends the final
SIGKILL to whatever remains in the cgroup once the main process exits, and
`process` only encodes accidental orphaning, which systemd itself discourages.
An earlier draft of this entry proposed `mixed` and was wrong.
The split:
```text
orchestra-worker.service
└── orchestra-worker
orchestra-tmux.service
└── tmux -L orchestra
└── claude sessions
```
`deploy/orchestra-tmux.service` is new. `deploy/orchestra-worker.service` gains
`After=`/`Wants=` on it, ordering only: a worker that finds the runtime missing
must report that rather than be stopped by it. The unit carries an idle
`orchestra-runtime` session so the server outlives its last agent pane.
Probed live on a throwaway socket: `tmux -L <s> new-session -d` leaves the
server running under its own pid after the client exits, so `Type=forking`
resolves a main pid. `systemd-analyze verify` passes.
`User` must match between the two units, because the socket lives under
`/tmp/tmux-$UID`. The installed worker unit on workpc runs as `kami` while
`deploy/orchestra-worker.service` still says `orchestra`. The staged copy at
`~/orchestra-deploy/orchestra-tmux.service` is set to `kami` to match reality.
### What F28 retroactively explains
- Run 3's `no server running` at 11:02:16 followed the 10:45:26 worker restart.
- The same at 13:06:22.
- Run 4's refusal could not be delivered at 13:32:16, seconds after the
13:31:51 restart.
- F16's missing-pane branch has been firing on deployment, not on real
execution-runtime loss. Its proof stands as written, but the trigger was
self-inflicted.
### F28 proof, to run after the split
```text
1. start task and obtain live pane P
2. record pane/session identity
3. systemctl restart orchestra-worker
4. assert pane P still exists with identical agent identity
5. new worker starts
6. worker reconciles lease + pane P
7. no TaskReleased / new lease caused merely by worker restart
8. agent continues under the same lease epoch
```
Then separately, to restore the meaning of F16's missing-pane branch:
```text
restart orchestra-tmux
→ pane disappears
→ F16 refuses renewal
→ lease expires and requeues
```
### Run 4 diagnostic continuation, partial
The chain ran and failed only at the last step, at 13:32:16:
```
deliver phase refusal 06G46P6KE25Y04VVF7VRZMHZ78: tmux send-keys ... -l -- Orchestra refused your phase request: phase request refused: task 06G46P6KE25Y04VVF7VRZMHZ78 may only move to "research", not "implement"
...: no server running on /tmp/tmux-1000/orchestra: exit status 1
```
Live-proven by that one line: F25, the boundary ran on claude at all; F21's
reading half; F26's premise, the coordinator naming the one valid target; F27,
the 409 classified and routed to `sendPrompt`. The request file survived the
failed send, which is the correct branch.
Not proven: the receipt itself, because F28 had already destroyed the pane.
+42
View File
@@ -0,0 +1,42 @@
# The tmux execution runtime, owned separately from the worker.
#
# The worker used to spawn this server implicitly on its first tmux command,
# which put the server and every agent pane inside the worker unit's cgroup.
# Restarting the worker then destroyed the sessions it was restarting to
# manage: a deploy killed the run it was deploying for, and F16's missing-pane
# branch fired on deployment rather than on real execution-runtime loss.
#
# KillMode does not fix that. Under `mixed` systemd still sends the final
# SIGKILL to whatever remains in the cgroup, and `process` only encodes
# accidental orphaning. The runtime has to be a service of its own.
#
# Lifecycle this gives you:
# systemctl restart orchestra-worker -> panes survive, worker reconciles
# systemctl stop orchestra-tmux -> panes die, deliberately
#
# User MUST match orchestra-worker.service. The socket lives under
# /tmp/tmux-$UID, so a mismatch leaves the worker talking to a different
# server, or to none.
[Unit]
Description=Orchestra tmux execution runtime
After=network.target
[Service]
Type=forking
User=orchestra
# The socket name is the worker's tmux_socket for this harness (-L, not -S).
# Keep the two in step; the worker does not create the server any more.
#
# The idle session exists so the server outlives its last agent pane. Without
# it the runtime exits whenever a task completes, and the next launch spawns a
# fresh server back inside whichever cgroup asked for it.
ExecStart=/usr/bin/tmux -L orchestra new-session -d -s orchestra-runtime
ExecStop=/usr/bin/tmux -L orchestra kill-server
# The runtime coming back empty is honest: the panes are gone either way, and
# F16 observes that as real execution loss. Refusing to restart would instead
# block every later launch until an operator noticed.
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
+5
View File
@@ -2,6 +2,11 @@
Description=Orchestra federation worker
After=network-online.target
Wants=network-online.target
# Ordering only, deliberately not Requires. The execution runtime owns the
# agent panes and must outlive a worker restart, and a worker that finds it
# missing has to report that rather than be stopped by it.
After=orchestra-tmux.service
Wants=orchestra-tmux.service
[Service]
Type=simple