orchestra/BURNIN.md

Burn-in: live conformance, not more architecture

Written 2026-08-26. The workflow is feature-complete enough to exercise. What remains is empirical: run the same real task shape through each harness and classify what breaks. Isolated tests are insufficient by design here, and only the live owner path establishes conformance.

Do not add workflow features while this is running. Findings decide the next implementation work.

Burn-in build identity

77a2b323fabcf080d7542061ae2d7b3c34eef5b7

Both halves must report exactly this revision before a task is created. Neither needs a credential now: the coordinator prints it in docker logs orchestra-api and the worker in journalctl -u orchestra-worker. The same object is at GET /v1/admin/diagnostics and GET /v1/federation/workers behind the operator login.

Build both with deploy/build.sh <outdir>, which refuses a dirty tree. Later documentation-only commits do not change this identity, so a rebuild either passes this revision explicitly or accepts the new one and redeploys both halves. Never one half.

Deployment state, 2026-08-26 18:35

Half	State
Coordinator (homesrv)	Deployed at 6f9300b. Rebuilt with --build-arg BUILD_REVISION, recreated, /readyz ready, self-reporting the revision in its log
Worker (workpc)	Staged, not installed. ~/orchestra-deploy/orchestra-worker sha256 2b1c43071eaf6b5c15b110de39f204038a9629897e0ce3dacf45be96a6e6529e. Installed binary is still the 2026-07-30 build

Two operator steps remain, both needing root:

sudo install -m 0755 /home/kami/orchestra-deploy/orchestra-worker /usr/local/bin/orchestra-worker
sudo systemctl restart orchestra-worker
journalctl -u orchestra-worker -n 5 --no-pager   # must print revision 6f9300b...

A third blocker, found while preparing the pane check:

herdr is not running on workpc. herdr status server reports not running, the socket at /home/kami/.config/herdr/herdr.sock refuses connections, and its log stops at 2026-07-30. workpc-opencode therefore cannot start a pane, even with the worker installed. The worker logs serving harness workpc-opencode (opencode) on herdr backend at startup without touching the socket, so that line is not evidence the backend is reachable. herdr is an interactive terminal workspace manager: running herdr in a terminal launches or attaches to the persistent session and starts the server. Confirm with herdr status server before creating a task.

Then scrub the pane environment before letting an agent run.

Where the pane environment actually comes from

The two workpc harnesses inherit different environments, so one scrub does not cover both.

• workpc-opencode (herdr backend). The pane is created by the herdr daemon, which is a separate long-running process. It inherits herdr's environment, not the worker's. Scrubbing worker.env does nothing here. Whatever environment herdr is started with is what every opencode pane gets.
• workpc-claude (tmux backend). TmuxBackend.StartAgent runs tmux new-session through exec.CommandContext with no Env set (internal/herdr/tmux.go). If the tmux server is not already up, the worker starts it, and that server inherits the worker's full environment. Every claude pane then inherits it too.

This exposes a conflict the scrub alone cannot resolve. The worker legitimately needs ORCHESTRA_WORKER_TOKEN (or ORCHESTRA_WORKER_TOKEN_<ID>) and ORCHESTRA_FEDERATION_ADMIT_TOKEN, and deleting them breaks the worker. Keeping them means the tmux path hands them to the agent. Closing it needs either a filtered cmd.Env when the backend spawns a pane, or a tmux server started separately with a clean environment. Not on flow 1's path, because flow 1 is opencode only. Close it before step 6.

Baseline: one harness, one path

workpc-opencode only. Codex is a coverage gap, not a prerequisite, and no worker runs on homesrv. Prove one real path first:

real vikunja task
→ workpc-opencode
→ frame/research/plan
→ trajectory gate if configured
→ implement
→ independent review
→ task pr
→ human merge
→ completed

Then raise difficulty in this order:

1. opencode boring success
2. opencode mid-session human correction
3. opencode forced rotation
4. opencode pr rejection -> fix -> merge
5. opencode reconcile outage/recovery
6. representative cases on claude
7. add the codex adapter/config
8. cross-harness rotation: opencode -> claude/codex

Herdr transports, so nobody misdiagnoses again

No entry in the live config.jsonc sets backend or address, so all six resolve to <machine>:9245 (registry.defaultHerdrPort). Both ports are closed. That says nothing about the workpc harnesses: they are worker-owned, and the coordinator logs worker-owned on workpc; coordinator probe skipped for each. The worker reaches workpc-claude over a tmux socket and workpc-opencode over /home/kami/.config/herdr/herdr.sock. Do not diagnose harness availability from a TCP probe of 9245.

Evidence to record per run

One row per run. The launch instruction is now dumped to <worktree>/.orchestra/launch.md (herdr.LaunchContextFile) at every launch, local and federated, so the context is auditable without reading pane scrollback.

harness            (claude | codex | opencode)
task id
lease epoch        (every epoch, if the task rotates)
session ids        (pane ids per epoch)
human decision ids
launch context     (sha256 of .orchestra/launch.md, per launch)
handoff ref
git sha            (at every boundary: launch, each handoff, review, submit, merge)
review ref
submission ref
pr id
completion receipt

The five flows

1. Boring success. task → research → plan → implement → review → pr → merge.
2. Mid-session correction. Agent is doing A, the human says B, the next verified boundary delivers B, and A becomes history rather than a competing instruction.
3. Rotation. Agent A hands off, agent B resumes. Switch harnesses across the boundary if both are up.
4. Human rejection. Submit sha A, comment on the pull request, task reopens, fix at sha B, a fresh review, the same pull request, merge.
5. Failure path. Human source goes down, the reconcile streak escalates, the session hands off, the successor lease is refused, the source recovers, and a corrected successor starts.

What to inspect by hand

For every launch, read .orchestra/launch.md and ask only:

does this agent know
- what the task actually wants?
- what was most recently decided?
- what phase it is in?
- what earlier material is merely historical?
- what to do next?

Then compare that against what the model did.

Classify before fixing

Every failure gets exactly one label before any code is written:

authority bug            the wrong thing outranked the right thing
context-selection bug    the renderer showed or hid the wrong material
lifecycle bug            state, lease, phase or event handling is wrong
adapter/harness bug       the pane, occupancy, boundary or prompt path is wrong
model-following failure  the context was right and the model ignored it
operator-policy gap      credentials, deployment or configuration

This matters because the cheap response to every failure is another prompt rule, and prompt rules accumulated to compensate for lifecycle or adapter bugs are how the enforcement boundary rots. A model-following failure is the only class a prompt change should ever answer.

Pane credential cleanup (operator, in parallel)

The agent pane inherits the worker's environment. On workpc the worker is orchestra-worker.service, User=kami, EnvironmentFile=/etc/orchestra/worker.env (mode 0600, root-owned, deliberately unreadable here). Remove from that file, and from anything else the pane inherits:

gitea mutation token
vikunja mutation credentials
orchestra operator / system token

Retain only ORCHESTRA_AGENT_TOKEN and credentials a specific task genuinely needs. If an agent needs a forge operation, it asks Orchestra to perform it.

Then verify from inside a real pane:

git push                 # must fail unless Orchestra supplied the auth
curl .../v1/tasks/<id>/submission   # must be 403 for the agent surface
tea pr create            # must have no usable mutation credential

Note what is not enforced by code: nothing in this repo scrubs the pane environment, and a fake git earlier in $PATH is not enforcement because /usr/bin/git bypasses it. Credential isolation is the enforcement. Confinement of network, filesystem and destructive commands belongs to whatever launches the process (herdr, a container, systemd, bubblewrap), not to Orchestra, which supplies identity and policy inputs.

Exit criterion

Roughly 10 to 20 real tasks, with the five flows covered on each harness that is actually up. Then read the classified failures and decide the next implementation unit from them.

Live findings, 2026-08-26 21:35

Both halves are paired at 6f9300b, and herdr is up (0.7.5, protocol 17). What remains before flow 1, and what the in-pane credential checks will really show.

Flow 1 has no repository to run in

workpc-opencode declares exactly one project, test-e2e, pointing at /tmp/test-e2e and /tmp/test-e2e-worktrees. Neither path exists on workpc after today's reboot, and /var/lib/orchestra/repos does not exist there at all. Nothing can be leased to this harness until it has a project backed by a durable local repository.

The only configured task source is Gitea kami/correx at https://gitea.kvmx.ru (ORCHESTRA_GITEA_URL/OWNER/REPO), so a real task means a correx issue. correx already has machine_affinity: ["homesrv","workpc"] in the coordinator's config.jsonc. What is missing is a workpc-local entry:

// /etc/orchestra/worker-projects.json, on workpc
{
  "correx": {
    "repo": "/home/kami/orchestra/repos/correx.git",
    "worktree_root": "/home/kami/orchestra/worktrees/correx",
    "remote": "origin"
  }
}

Not /tmp. The burn-in outlives a reboot.

There is no vikunja ingest

This repo has no Vikunja provider. /readyz reports gitea and jsonl. A "real vikunja task" cannot enter Orchestra today: it arrives as a Gitea issue or through the JSONL watcher.

git push from a pane will succeed, and no Orchestra setting stops it

Corrected 21:45. The mechanism is SSH, not HTTPS. HTTPS has no stored credential on workpc, where git push over https:// fails with could not read Username. Pushing works over ssh://git@gitea.kvmx.ru:2222 using kami's default RSA identity, which Gitea lists as the key named workpc. It carries no passphrase, so no agent is needed.

That identity is what lets the worker push, and panes run as the same user with the same home directory, so an agent inherits it. worker.env is irrelevant to this. Real isolation needs panes under a different unix user. Expect git push --dry-run to succeed from a pane, and record it as an operator-policy gap rather than a code defect.

The worker inherits ambient Git credentials by design: git() in cmd/orchestra-worker/main.go runs exec.CommandContext with no environment of its own. There is no separate credential path for the worker to hold something the pane does not.

The agent surface is currently unauthenticated

ORCHESTRA_AGENT_TOKEN is unset on the coordinator, and an unset surface token means the middleware performs no check for that surface. Probed live:

POST /v1/tasks/<id>/decision-request  -H 'X-Orchestra-Surface: agent'  -> 404 (reached the handler)
POST /v1/tasks/<id>/phase             -H 'X-Orchestra-Surface: agent'  -> 403 (refused)

The capability boundary holds. Authentication does not. Set ORCHESTRA_AGENT_TOKEN in the coordinator .env before the burn-in, and note the same is true of ORCHESTRA_MCP_TOKEN and ORCHESTRA_MAVEN_TOKEN, both unset.

Two more unset settings that change burn-in behaviour

• ORCHESTRA_FEDERATION_ADMIT_TOKEN is unset, so any caller may register a new worker identity. Existing identities stay protected, because registering an existing id with a different token is refused.
• ORCHESTRA_REVIEW_ACTORS is unset, which human.Trust reads as "anyone not explicitly ignored". Flow 4 will accept a task-moving comment from any Gitea actor. Set it to kami.

Flow 1 setup, 2026-08-26 21:45

Target: kami/test-e2e on Gitea, a throwaway repo, rather than correx.

Done:

• Ingest switched. ORCHESTRA_GITEA_REPO=test-e2e in the coordinator .env (previous file kept as .env.pre-burnin-20260826), container recreated, still reporting 6f9300b. The project id must equal the repo name, because main.go sets Project: os.Getenv("ORCHESTRA_GITEA_REPO"). test-e2e already exists in config.jsonc with machine_affinity: ["workpc"].
• Durable repo on workpc. Bare clone at /home/kami/orchestra/repos/test-e2e.git, worktree root /home/kami/orchestra/worktrees/test-e2e. Origin points at ssh://git@gitea.kvmx.ru:2222/kami/test-e2e.git, and git push --dry-run reports Everything up-to-date. The old /tmp paths did not survive the reboot and must not come back.

Remaining, needs root:

// /etc/orchestra/worker-projects.json
{
  "test-e2e": {
    "repo": "/home/kami/orchestra/repos/test-e2e.git",
    "worktree_root": "/home/kami/orchestra/worktrees/test-e2e",
    "remote": "origin"
  }
}

Then sudo systemctl restart orchestra-worker.

Store state, checked before the first run: 30 tasks, all test-e2e, with 23 blocked, 6 completed and 1 failed. All are July leftovers and none holds a lease. The coordinator has run ResumeAnsweredBlockers every second for hours without resuming any of them, so they are inert rather than merely quiet. The July stuck task 06FT6CKD9Y98AZRX6X8K3QXFZG is now failed.

A stale branch orchestra/scratch/oc-06ftgkjadcd2hwjn2zwjen90q4 exists on the remote from an earlier run. Harmless, but it is not from this burn-in.

Run 1: task 06G3YR34117MAYT6KEAC9RJHD0, 2026-08-26

Evidence ledger:

harness         workpc-opencode (herdr backend)
task            06G3YR34117MAYT6KEAC9RJHD0
source          gitea:test-e2e/1  "Add a --version flag to the healthcheck script"
attempts        3 leases: two nacked, third launched
pane            wN:p1, agent oc-06g3yr34117mayt6keac9rjhd0
agent session   ses_fc096de31ffelZeMngQEyBTxfh
worktree        /tmp/test-e2e-worktrees/06G3YR34117MAYT6KEAC9RJHD0
branch          orchestra/06G3YR34117MAYT6KEAC9RJHD0
head at launch  5e6c4783d4213a934dc486160b888b6014bc2d03
launch context  .orchestra/launch.md, 2138 bytes
phase           frame
decisions       none

What worked

• The launch context is right. Authority order, the ambiguity ladder, the phase brief with frame: ... Do not change code, Current human decisions: None recorded, and a verified git state with worktree, branch and head.
• The agent respected the phase. agent_status: done, no code changes, only Orchestra's own scratch directory in the tree.
• A bad launch nacked cleanly. The first attempt failed on the base checkout and the task went back to queued with lifecycle_phase: launch_nacked and attempt: 1. No orphan pane, no stuck lease.

Fixed during the run

• F1, authority bug, 2753a8d. Every federated launch died with effective intent: federation: 401 Unauthorized: unauthorized surface. GET /v1/tasks/<id>/intent exists for workers, and the authz worker-path exemption never included it. Twenty test packages passed throughout.
• F2, authority bug, 09e572f. The rendered goal was the issue title alone and acceptance read Not stated., because Gitea.event parsed the issue body and dropped it from the TaskCreated payload. The store already read description. Every Gitea-sourced task so far ran on its title.
• F3, self-inflicted, a0209a2. The launch dump left .orchestra/ untracked in a worktree whose own context said uncommitted changes: false. It would have polluted the gate, the review diff and the agent's git status.

Open, classified

• F4, adapter/harness. rotation ... activity degraded: activity unknown: adapter: resolve session file: adapter: harness "opencode" has no session-file resolver. Thrash and activity triggers are permanently degraded on opencode. Observable in the worker's last_error, non-fatal.
• F5, lifecycle. The router never leased this task. Every gate checks out by hand: worker online: true, herdr_status: reachable, fresh checked_at, supported_projects: ["test-e2e"], quota empty, no active leases, capability empty. A direct POST /v1/tasks/<id>/lease succeeded instantly. The refusal is upstream of Store.Lease and silent by construction (internal/router/router.go:197). All three runs were leased by hand.
• F6, observability. Worker health reports active_task: null while the store shows the task leased and herdr shows a live opencode agent in the task's worktree.
• F7, operator policy. ORCHESTRA_TUI_TOKEN is unset, and an unset surface token means no authentication for that surface. That is how the manual leases above were issued, unauthenticated, from another machine. The TUI surface is FullControl, so this is the whole control plane, not just the two agent request endpoints. Set the token.
• F8, correctness. human.Reconciler.Reconcile iterates every configured source for every task, so the three queued correx tasks have their external ids looked up in kami/test-e2e. A source should only reconcile the tasks that came from it. Likely why those three never lease.

Fix pass before run 2, 2026-08-26 23:50

Burn-in identity is now 77a2b32. The coordinator is deployed at it. The worker is staged at it, sha256 2a850f2d1102d390..., still needing root to install.

Closed

• F7, security. authz.RequireCredentials refuses startup when a full-control surface has no token, rather than logging it. ORCHESTRA_TUI_TOKEN is set in the coordinator .env. Verified live: an unauthenticated POST on the TUI surface now returns 401. Web is exempt because Sessions makes its login mandatory. ORCHESTRA_MCP_TOKEN, ORCHESTRA_MAVEN_TOKEN and ORCHESTRA_AGENT_TOKEN remain unset, so those surfaces are still unauthenticated for reads and for their three request endpoints. Bounded by capability, worth closing, not startup-fatal.
• F5, lifecycle. Every eligibility gate now records a router.Rejection, exposed at GET /v1/router/health and reset per pass. No gate was weakened. The live output immediately explained the three stuck correx tasks: worker has not declared project correx. A queued task in retry backoff was skipped before the candidate loop and recorded nothing at all, which is the shape that hid the original case; it now reports retry backoff until <time>.
• F8, correctness. Reconciliation is bound to task.Source, the provider:project identity the ingest stamped. A source that cannot prove it owns the task is skipped, and a task with no matching source reconciles to nothing and still launches. The integration fixture had encoded the bug: it ingested from jsonl and reconciled from gitea.
• F11, outage. The API entered a restart loop exiting with invalid event: until_ns required. ValidateEvent compared until_ns against time.Now() for TaskLeased and TaskLeaseRenewed, so a lease event that was valid when written failed validation once it expired. store.Open replays the tail after the snapshot and log.Fatals on the first invalid event, so the coordinator refused its own history. Validation of a durable event is now time-independent. Latent since the field existed: it needed a renewal in the post-snapshot tail plus a restart after that renewal expired.

Also found

• F9, lifecycle. An operator cannot release a leased task through POST /v1/tasks/<id>/release without knowing its harness_id and lease_epoch, because Store.Append fences lifecycle events on a leased task and the endpoint passes the request body through unchanged. Correct that only the owner releases, but there is no operator escape hatch.
• F10, hygiene. A coordinator-side release under a live worker leaves the worker renewing a lease it no longer holds. Combined with F11 that produced a per-second invalid-event log line.

Shared checkout

Another session is working on the auth and frontend layers in this same checkout. Two consequences, both handled: deploy/build.sh and the container image now build in a detached worktree of the revision they stamp, so no uncommitted work is compiled into a stamped binary and no in-progress edit blocks a deploy. Commits from this session name their paths rather than using git add -A. The first commit, 7f12c7f, predates that discipline and swept in whatever was uncommitted at the time, including web/src and internal/authn.

Run 1 is closed as a diagnostic

Task 06G3YR34117MAYT6KEAC9RJHD0 was released with reason abandoned diagnostic run, supplying the lease fence by hand per F9. It sits at attempt: 3 and the router will fail it on its next pass. It is not a conformance run: it needed three manual leases and its authority was built before the issue-body fix.

Run 2 preconditions

worker installed at 77a2b32 and restarted
coordinator at 77a2b32                     done
F7 closed                                  done
F5 exposed                                 done
F8 fixed                                   done
fresh issue, no manual leasing

Runs 2 and 3, 2026-08-27

Written from live evidence. Neither run was a conformance pass, and both were worth more than one: run 2 found three blocking bugs behind each other, and run 3 found two more plus the reason six isolated probes disagreed with production.

AUDIT.md is uncommitted and owned by another session, so the burn-in ledger lives here.

Fixed, with the revision each landed in

• F12, lifecycle, 0d67af9. Store.QuotaSince reported an empty window as unknown, QuotaAvailability fails closed on unknown, and every herdr in config.jsonc declares a quota limit. The only producer of a receipt is a completed lease, so nothing could ever be leased. An empty window is now observable zero. A receipt that declares its own consumption unknown still fails closed. Verified live: run 2 leased three seconds after the fixed coordinator started.
• F13, observability, 0d67af9. federatedAvailability restated a quota refusal as worker health, so router health said stale heartbeat against a heartbeat one second old. Gates name themselves through router.ReasonedAvailability. Verified live.
• F14, authority, a5d361b. Gitea ingest never set acceptance. One recognized heading, bullets and checkboxes until the next heading, order preserved, section removed from the description. Verified live on run 3: five ordered items, prose above the heading kept as the description.
• F15, adapter, 1fd82f8. Two bugs. TmuxBackend.Prompt wrote the whole instruction with send-keys -l, Claude Code coalesced it into a paste, and the Enter was absorbed. And TaskLaunchAcknowledged meant "Prompt returned nil", not "the harness accepted it". Launch transport became a backend property, and ConfirmLaunch began polling for proof. The detection works. The fix did not: see F17.
• F16, lifecycle, 1888d42. renewLeases renewed whenever a session existed and PaneCapture succeeded, so a pane that opened and never started renewed forever. This is why the July task stayed orphaned: F15 explains why nothing started, F16 why the lease never let go. Renewal now needs the agent busy, or the pane capture to differ from the hash recorded at the previous renewal. lease.ProgressSHA carries that hash.
• F17, adapter, f54fb00. pendingInput scanned every line beginning with the prompt marker, but queued and already-accepted input renders with the same prefix. Only the editor owning the pane cursor is unsubmitted, so inputState reads #{cursor_y} and captures screen rows without -J, which would invalidate the row index. On that footing ConfirmLaunch became an active submit protocol: resend Enter while the live editor still holds exactly what was submitted, at most three times, no closer than two poll intervals, then observe until the deadline. Queued input confirms rather than fails. Evidence records confirmation, submit_attempts and both timestamps.
• F19, lifecycle, edff021. A task that reached the router's MaxAttempts was permanently terminal. TaskReleased only increments Attempt, TaskCorrected could not touch it, and no HTTP route emitted a correction. POST /v1/tasks/{id}/retry requires the task to be failed, unleased, and failed with reason: retry_limit, then appends one TaskCorrected naming that failure with state: queued and attempt: 0. Identity, goal, acceptance, decisions, phase and artifact refs all survive, and the original failures stay in the log. operation_id is required and makes it idempotent. Deliberately not a generic correction endpoint.

The finding that mattered most

Run 3 failed three times with prompt_not_submitted, and six isolated probes of the same code path could not reproduce it: fresh session, untrusted directory with the trust dialog, launch.md present before startup, 100ms readiness polling, env -i with only the worker's systemd variables, and the real StartAgent/Prompt/ConfirmLaunch against a byte-identical git worktree. All six submitted on the first Enter.

F17 turned that disagreement into data instead of a theory. The first live launch after it deployed:

launch 06G44JZB80MZBEY97196EZN8EC confirmed: confirmation=editor_cleared
submit_attempts=2 first_submit_at=2026-08-27T09:12:00.801855937Z
confirmed_at=2026-08-27T09:12:01.315587139Z

Production needs the second Enter. Isolated probes need one. The submit is not deterministic, which is exactly what F15's ConfirmLaunch comment asserted it was.

Still open

• F18, observability. A queued launch used to be misread as unsubmitted. F17 fixes the predicate, but nothing else that parses pane text distinguishes the active editor from history, so the same class of error can recur wherever PaneCapture output is matched.
• F9 is unchanged by F19. An operator still cannot release a lease someone else owns without supplying that owner's harness_id and lease_epoch, which means reading them out of the event log first. F19 recovers a terminal task, which is a different escape hatch.
• F6, observability. Worker health reported active_task: null while the store showed the task leased. Run 2 showed this was partly honest, because the worker really had no working agent. Recheck now that launches confirm.
• F4, adapter quality. harness "opencode" has no session-file resolver, so activity and thrash triggers are degraded on opencode.
• F10, hygiene. A coordinator-side release under a live worker leaves the worker renewing a lease it no longer holds.
• Unset gated tokens: ORCHESTRA_MCP_TOKEN, ORCHESTRA_MAVEN_TOKEN, ORCHESTRA_AGENT_TOKEN.
• gofmt -l fails on internal/provider/provider.go, internal/router/router.go and internal/webui/webui.go at edff021. None were touched by these runs. internal/router/router.go picked it up in 0d67af9. go vet passes, which is why nobody noticed.

Operator actions taken by hand, and why

• Task 06G3ZCZWJ3QHF992ZMDGSJ0PYG, run 2, was blocked with block_reason: operator_block rather than released. A released task returns to queued, and the router leases queued tasks, so releasing it would have put it in competition with run 3.
• The first block attempt returned task version conflict. Store.validateTransition fences every lifecycle event on a leased task, so the payload needs harness_id and lease_epoch even though the HTTP handler does not ask for them. That is F9 in practice.
• Task 06G44JZB80MZBEY97196EZN8EC, run 3, was recovered with F19's retry rather than by filing a fourth issue. That keeps kami/test-e2e#3 served by its own task and proves terminal recovery in the same run.

Deployment boundary

Both halves report edff021, built from a detached worktree of that revision. Worker sha256 68ac265455acb0e007bfb0e898a91b317cd7289d7f2bf5da6df1224e1d6706a5 at /usr/local/bin/orchestra-worker. Coordinator image built 13:08:30 +0400.

Run 3: autonomous expiry and relaunch, 2026-08-27 11:02 UTC

Task 06G44JZB80MZBEY97196EZN8EC, issue kami/test-e2e#3. Observed live, no operator action in the window. The stranded epoch was 06G44ZBZX4YH6PRN7Y4ZH8GG3W.

seq	at (UTC)	event	epoch
358	10:32:09	TaskLeaseRenewed v15	06G44ZBZX4YH6PRN7Y4ZH8GG3W
359	11:02:16	TaskReleased v16, reason: lease_expired	06G44ZBZX4YH6PRN7Y4ZH8GG3W
360	11:03:17	TaskLeased v17	06G45RVKAW20H1D22NHBRYKMN8
361	11:03:22	TaskLaunchAcknowledged v18, lifecycle_phase: started	06G45RVKAW20H1D22NHBRYKMN8

Expiry to acknowledged launch: 66 seconds, autonomous. Every event from seq 353 onward carries surface: system. The last tui event is the TaskCorrected at 09:11:55, before the window.

Launch receipt, worker journal, 15:03:22 +04:

launch 06G44JZB80MZBEY97196EZN8EC confirmed: confirmation=editor_cleared submit_attempts=2 first_submit_at=2026-08-27T11:03:22.203859355Z confirmed_at=2026-08-27T11:03:22.717657436Z

F16 is partial, not closed

The hard predicate passed: no renewal carried the stranded epoch after 10:45:26. The run did not exercise the progress comparison. Worker health records why the renewal was skipped:

"last_error": "validate lease 06G44JZB80MZBEY97196EZN8EC: tmux capture-pane -p -J -S -200 -t =orchestra-06g44jzb80mzbey97196ezn8ec-e53607ea:1.0: no server running on /tmp/tmux-1000/orchestra: exit status 1",
"error_at": "2026-08-27T11:02:16.213318366Z"

That is the paneProgress error path at cmd/orchestra-worker/main.go:973, not the default branch at line 992. The tmux server was gone, so no pane text could be hashed.

• Proven: dead or missing pane, no renewal, lease expires, autonomous re-lease and relaunch.
• Unproven: live pane with an idle agent and unchanged output, renewal refused.

Epoch reset, stated precisely

The new epoch and its acknowledged launch establish the reset structurally. leases in /var/lib/orchestra/worker-state/workpc-claude.json holds only {epoch, version, until}, with progress_sha absent under omitempty. No runtime value of ProgressSHA, UsageBaseline or PickupAcknowledged was observed.

F6 closed

Worker health reports active_task_id: 06G44JZB80MZBEY97196EZN8EC with active_pane_id set, against a leased task.

F18 sharpened, non-blocking

recordError (cmd/orchestra-worker/main.go:70) keeps one slot, so the earlier renewal decision near 10:52 was overwritten and cannot be read. One last_error slot cannot preserve a causal sequence. Later replacement: a bounded recent-error ring or event-backed observations, roughly the last 8 {at, class, message} entries. Do not land this during a live run.

Ledger after this checkpoint

F6   closed
F14  closed
F15  closed by detection, transport fix pending live proof
F16  partial: missing-pane branch proven, static-live-pane branch unproven
F17  fixed, isolated live proof
F18  open observability
F19  fixed
F20  awaiting first post-frame confirmed input

Next decisive checkpoint, F20

frame completes
→ Orchestra emits the next input
→ worker journal must contain: input to <pane> confirmed: confirmation=<kind> submit_attempts=<n>
→ only then may the agent continue

Receipt appears and the model proceeds: let the run continue through phase transitions. Input lands with no receipt: F20 fails. No input attempted after frame: a different lifecycle or phase-advance bug.

Run 3 conformance result, 2026-08-27 12:00 UTC

run 3 conformance result: failed
cause: no autonomous phase-advance path

F16 closed

missing-pane branch: proven
live-pane idle + unchanged output branch: proven
external editor input ignored as progress: proven

Second branch evidence, worker health at 11:53:46 with the pane alive and herdr_status: reachable:

"last_error": "lease 06G44JZB80MZBEY97196EZN8EC not renewed: agent status idle and pane unchanged since the last renewal",
"error_at": "2026-08-27T11:53:46.195553513Z"

That is the default branch at cmd/orchestra-worker/main.go:992.

F21, lifecycle: the phase brief promises a protocol that does not exist

.orchestra/launch.md:62 tells the agent Orchestra decides when this phase ends. Ask for a phase change, do not declare one. The agent complied and printed Nothing blocks. Ready for a phase change to implement.

phase brief:  "ask for a phase change"
agent:        asks
worker:       has no representation of that request
coordinator:  AdvanceWorkPhase exists, but nothing invokes it autonomously

operations.AdvanceWorkPhase is reachable only from the HTTP handler at cmd/orchestra/main.go:883 and from internal/operations/review.go:101. The worker's only post-frame send is a decision notice at cmd/orchestra-worker/main.go:1639, gated on len(answer.Decisions) == 0 returning early.

The fix is a bounded agent intent, not pane-text matching:

type PhaseAdvanceRequest struct {
    From WorkPhase
    To   WorkPhase
}

verified turn boundary
→ obtain bounded phase intent from harness
→ validate requested transition
→ submit to coordinator
→ operations.AdvanceWorkPhase
→ if transition requires sealed artifact:
     collect/seal artifact first
→ rotate/start successor if phase policy requires fresh context

frame needs no artifact. research and plan take the same path but must seal their artifact before AdvanceWorkPhase accepts them.

F22, lifecycle: nothing reacts to WorkPhaseChanged at runtime

t.WorkPhase is read only when a launch context is built, at cmd/orchestra-worker/main.go:369 and internal/orchestrator/orchestrator.go:1218. No code path rotates, relaunches or notifies a live session when the phase changes. A manual advance therefore takes effect only at the next launch, and leaves the current idle pane idle.

F23, lifecycle: a decision cannot exist before a submission

answer.Decisions is the only input sendPrompt ever carries post-frame. Decisions come from operations.ReflectSubmission, and human.PullRequestState.FeedbackAfter returns input only strictly after a submission. A task still in frame has no submission, so no decision can be recorded for it.

Consequence: F20 cannot be exercised on this task in its current state. A manual phase advance does not produce a post-frame input, because nothing reacts to the phase change and no decision can exist yet.

Observation kept separate from F20

❯ go ahead and implement it appeared in the editor again on attempt 2, cursor at cursor_x=2, never submitted. The agent transcript at ~/.claude/projects/-tmp-test-e2e-worktrees-06G44JZB80MZBEY97196EZN8EC/97b30e4d-90a7-49bc-a9b8-7d9e450831e4.jsonl holds exactly one user message, the launch prompt at 11:55:52.236Z. No worker send and no receipt exist for it. Origin unknown, external to Orchestra. F20 does not absorb it.

Ledger after this checkpoint

F6   closed
F14  closed
F15  closed by detection, transport fix pending live proof
F16  closed, both branches live-proven
F17  fixed, isolated live proof
F18  open observability
F19  fixed
F20  blocked, not merely awaiting: see F23
F21  open lifecycle, no autonomous phase-advance path
F22  open lifecycle, no runtime reaction to WorkPhaseChanged
F23  open lifecycle, no decision path before a submission

F21, F22, F23 implemented, 2026-08-27

Run 3 was not shepherded further. Attempt 3 was left unspent: F16 is closed on both branches, and another idle phase would have proven nothing new.

F23 was already implemented, and the earlier entry was wrong

human.Reconciler.Reconcile imports issue comments from task.Source and records HumanDecisionRecorded with no submission involved. It is wired at two points in cmd/orchestra/main.go: Store.PreLease, and Coordinator.ReconcileHumanInput, which RemoteTurn calls at every verified turn boundary. provider.GiteaComments.FetchAfter reads the task's own issue.

The earlier F23 entry traced EventHumanDecisionRecorded through internal/operations only and concluded decisions required a submission. That was a scoping error in the search, not a gap in the code. ReflectSubmission is the pull-request path and stays narrow; it was never the general decision source.

Consequence for the record: F20 had a legitimate post-launch send path throughout run 3. A comment on kami/test-e2e#3 would have produced a decision, a DecisionNotice at the next boundary, and a receipt.

F23 is closed as already-implemented, with tests added for the boundary it turns on.

F21, the phase-request protocol

The agent asks with a bounded file. Prose is not a protocol, so nothing matches on pane text.

.orchestra/phase-request.json   {"from": "frame", "to": "research"}
.orchestra/research.json        sealed before leaving research
.orchestra/plan.json            sealed before leaving plan

At a verified turn boundary the worker validates what it can see locally: the phase the agent believes it is in, the legality of the transition, and the presence and decode of the artifact the phase must seal. It then calls POST /v1/federation/phase with the lease epoch and a derived operation id. The coordinator calls the existing operations.AdvanceWorkPhase.

• internal/operations/workphase.go: RequestWorkPhase, ErrPhaseRequest. Fences on lease epoch, refuses a stale phase belief, refuses any target but the project's next phase, idempotent per operation id.
• internal/federation/client.go: Client.AdvancePhase.
• cmd/orchestra/main.go: the /v1/federation/phase route.
• cmd/orchestra-worker/main.go: requestPhase, phaseArtifact, phaseRequestFile.
• internal/agentctx/agentctx.go: phaseRequestBrief renders the protocol under the sentence that used to promise it.

The operation id is derived, not random: phase:<task>:<epoch>:<from>:<to>. A redelivery after a lost response carries the same id, so the coordinator returns the first event instead of advancing twice.

F22, a phase change ends that cognitive session

herdr.Session now carries Phase, the phase the session was launched to run. At a turn boundary a session whose task has moved on is rotated with reason phase_changed, which is added to the closed handoff-reason set in internal/herdr/adapter.go and given its own wording in RequestHandoffReason.

One comparison covers both cases: a change this worker requested, and one an operator made through POST /v1/tasks/{id}/phase. Both leave the same evidence, a session built for a phase that is no longer current.

F20 hole found and closed while implementing F22

CLIAdapter.prompt called backend.Prompt and returned. Every rotation and handoff prompt therefore went out unconfirmed, which is precisely the failure F20 exists to catch. RequestHandoffReason is one of its callers, so the phase rotation would have inherited it.

Fixed at the shared call site rather than per caller: prompt now routes through InputConfirmer and logs the same input to <pane> confirmed: receipt the worker logs.

Tests

go build ./..., go vet ./... and go test ./... all pass.

• internal/operations/phase_request_test.go: full path with each artifact sealed, skipped phase refused, stale phase belief refused, stale epoch refused, missing operation id refused, redelivery idempotent, operation id recorded.
• cmd/orchestra-worker/phase_test.go: request accepted then session rotates, external phase change rotates, unsealed artifact refused locally, malformed artifact refused locally, sealed artifact travels with the request, refused request retried, decision notice stays undelivered until confirmed.
• internal/integration/phase_protocol_test.go: a pre-submission issue comment steers a live leased session and is not re-sent once delivered; the phase-request path seals and fences at the coordinator.
• internal/human/pullrequest_window_test.go: pull-request feedback ignores anything not strictly after the submission, and applies trust.
• internal/agentctx/agentctx_test.go: the brief names the request file, the request shape, and the artifact to seal.

Ledger

F6   closed
F14  closed
F15  closed by detection, transport fix pending live proof
F16  closed, both branches live-proven
F17  fixed, isolated live proof
F18  open observability, non-blocking
F19  fixed
F20  fixed, adapter hole closed; live proof pending run 4
F21  fixed, tests only
F22  fixed, tests only
F23  closed, already implemented; tests added

Run 4 starts from here. Both halves must be rebuilt and redeployed before it begins, and the deployment boundary recorded as usual.

Deployment boundary for run 4, 2026-08-27

Revision 1f5bf7e, both halves, built from a detached worktree of that commit.

Half	Evidence
Coordinator, homesrv container	docker logs orchestra-api, orchestra revision 1f5bf7e66e0c6dc8dc1db794ab91193c6c38e118 built 2026-08-27T16:35:37+04:00 dirty false, /readyz 200
Worker, workpc systemd	pending operator install

Worker sha256 c188ec78f7b11bd0f3b72da2c3544c5c9477b4505816b30bd0f243f6fa8032da, staged at ~/orchestra-deploy/orchestra-worker.1f5bf7e.

Both worker registrations must report 1f5bf7e before run 4 starts.

F24, correctness, dormant on the current topology

CLIAdapter.LeasePrompt (internal/herdr/adapter.go:161) calls backend.Prompt and returns without confirming. It is the last Orchestra-owned editor write outside the delivery guarantee.

Dormant, not fixed: the coordinator-local harness path cannot execute here. No local herdr is eligible, and Coordinator.adapterFor refuses a session owned by a non-local herdr. Run 4 exercises the federated worker path, whose launch delivery is already confirmed at cmd/orchestra-worker/main.go:454.

The eventual fix:

CLIAdapter.LeasePrompt
→ backend.Prompt
→ ConfirmInput
→ only then acknowledge launch

The comment in place there argues that waiting would turn a long first turn into a false lease failure. It is obsolete: ConfirmInput proves submission, not completion of the first turn.

Required before claiming coordinator-local harness conformance. Not required before this federated burn-in, and changing the revision now would add churn run 4 cannot observe.

The y/n approval paths stay separate

internal/herdr/adapter.go:703 and cmd/orchestra-worker/main.go:1086 write to a y/n dialog, not an editor. They need a capture-revision-aware confirmation protocol of their own. The editor confirmer would report nonsense on them, so they are deliberately outside F20 and stay that way.

Run 3

Left running and untouched. Whatever it does from here is additional diagnostic evidence. Run 4 does not wait on it.

Run 4: failed conformance, 2026-08-27

Task 06G46P6KE25Y04VVF7VRZMHZ78, issue kami/test-e2e#4, revision 1f5bf7e.

run 4: failed conformance

cause:
f25, claude rotationTick bypassed federatedTurn

evidence:
phase-request.json existed
no WorkPhaseChanged followed
no boundary error was emitted

additional defects:
f26 phase brief advertised invalid shortcut
f27 phase refusal was not delivered to agent

What run 4 did prove

The front half of the chain is clean, with no operator lifecycle intervention.

seq	at (UTC)	event
370	13:11:29	TaskCreated v1, 7 acceptance criteria
371	13:11:30	TaskLeased v2, epoch 06G46P6Q4EAPF7DFDBFBTA9RJC
372	13:11:33	TaskLaunchAcknowledged v3

Issue to confirmed launch: 4 seconds. F14 passes, against run 1 where ingest set no acceptance at all. F15 and F17 pass again with confirmation=editor_cleared submit_attempts=1.

The agent wrote its request 25 seconds after launch:

/tmp/test-e2e-worktrees/06G46P6KE25Y04VVF7VRZMHZ78/.orchestra/phase-request.json
{"from": "frame", "to": "implement"}

The artifact half of F21 works. The reading half never ran.

F25, lifecycle: the turn boundary was unreachable on claude

rotationTick returned early for this harness before reaching the boundary, and federatedTurn has exactly one call site, below that return.

if w.harness == "claude" {
	if err := w.advanceClaudeContextReset(ctx, id, s); err != nil { ... }
	return
}

Introduced in 7f12c7f. Consequences on the harness both burn-in runs used: no phase request could ever be read, and every human decision recorded against a live claude session went undelivered.

This retracts an earlier claim in this ledger. F23's decisions do reach RemoteTurn, but on claude the worker never asked, so run 3 had no post-launch send path either. The statement that a comment on #3 would have produced a receipt was wrong.

Fixed: claude runs the common boundary after its context reset. It still skips the occupancy state machine, because it owns its own rollover. A turn boundary is not a rotation.

F26, authority: the brief advertised an invalid shortcut

The deployed brief rendered Legal values for "to" from here: research, implement, listing every domain-legal move. The project's path allows only research. The agent took the shortcut, reasonably.

Fixed: the brief names one target and states that a wrong one comes back with the right one. The project's path is Orchestra's to know.

F27, lifecycle: a refusal never reached the agent

A refused request only reached w.recordError. The agent would rewrite the same rejected file at every boundary with nothing telling it why, which is the silent-loop shape this codebase keeps producing.

Fixed: federation.StatusError makes a 409 classifiable. A refusal is delivered through sendPrompt, so it travels under the F20 guarantee, and the request file is cleared. A transport failure keeps the file and tells the agent nothing, because it is not an answer.

Ledger

F6   closed
F14  closed
F15  closed by detection, transport fix pending live proof
F16  closed, both branches live-proven
F17  fixed, isolated live proof
F18  open observability, non-blocking
F19  fixed
F20  fixed, live proof still pending
F21  artifact half live-proven, reading half pending
F22  fixed, live proof pending
F23  closed as already implemented, undeliverable on claude until F25
F24  open correctness, dormant on current topology
F25  fixed, tests only
F26  fixed, tests only
F27  fixed, tests only

F28, deployment: the execution runtime shares the worker's cgroup

Worker restart destroys agent sessions because the tmux execution runtime shares the worker service cgroup. tmux must be independently lifecycle-managed.

The worker spawns the server implicitly on its first tmux command, so the server and every pane land in orchestra-worker.service:

orchestra-worker.service
├── orchestra-worker
└── tmux server
    └── claude panes

KillMode does not fix this. Under mixed systemd still sends the final SIGKILL to whatever remains in the cgroup once the main process exits, and process only encodes accidental orphaning, which systemd itself discourages. An earlier draft of this entry proposed mixed and was wrong.

The split:

orchestra-worker.service
└── orchestra-worker

orchestra-tmux.service
└── tmux -L orchestra
    └── claude sessions

deploy/orchestra-tmux.service is new. deploy/orchestra-worker.service gains After=/Wants= on it, ordering only: a worker that finds the runtime missing must report that rather than be stopped by it. The unit carries an idle orchestra-runtime session so the server outlives its last agent pane.

Probed live on a throwaway socket: tmux -L <s> new-session -d leaves the server running under its own pid after the client exits, so Type=forking resolves a main pid. systemd-analyze verify passes.

User must match between the two units, because the socket lives under /tmp/tmux-$UID. The installed worker unit on workpc runs as kami while deploy/orchestra-worker.service still says orchestra. The staged copy at ~/orchestra-deploy/orchestra-tmux.service is set to kami to match reality.

What F28 retroactively explains

• Run 3's no server running at 11:02:16 followed the 10:45:26 worker restart.
• The same at 13:06:22.
• Run 4's refusal could not be delivered at 13:32:16, seconds after the 13:31:51 restart.
• F16's missing-pane branch has been firing on deployment, not on real execution-runtime loss. Its proof stands as written, but the trigger was self-inflicted.

F28 proof, to run after the split

1. start task and obtain live pane P
2. record pane/session identity
3. systemctl restart orchestra-worker
4. assert pane P still exists with identical agent identity
5. new worker starts
6. worker reconciles lease + pane P
7. no TaskReleased / new lease caused merely by worker restart
8. agent continues under the same lease epoch

Then separately, to restore the meaning of F16's missing-pane branch:

restart orchestra-tmux
→ pane disappears
→ F16 refuses renewal
→ lease expires and requeues

Run 4 diagnostic continuation, partial

The chain ran and failed only at the last step, at 13:32:16:

deliver phase refusal 06G46P6KE25Y04VVF7VRZMHZ78: tmux send-keys ... -l -- Orchestra refused your phase request: phase request refused: task 06G46P6KE25Y04VVF7VRZMHZ78 may only move to "research", not "implement"
...: no server running on /tmp/tmux-1000/orchestra: exit status 1

Live-proven by that one line: F25, the boundary ran on claude at all; F21's reading half; F26's premise, the coordinator naming the one valid target; F27, the 409 classified and routed to sendPrompt. The request file survived the failed send, which is the correct branch.

Not proven: the receipt itself, because F28 had already destroyed the pane.

F28 closed, and three defects it uncovered, 2026-08-28

The unit split is installed on workpc and F28 is proven live in both directions. Three new defects, F29, F30 and F31, were found while setting the proof up. None was visible in tests.

The tmux unit, as installed

The staged Type=forking draft was wrong in a way the earlier entry did not catch: without PIDFile=, systemctl show orchestra-tmux -p MainPID returned 0. Systemd had no main process to watch, so a crashed server would read as a clean exit and Restart=on-failure would never fire.

tmux -D runs the server in the foreground, which gives systemd the real pid. It also turns exit-empty off, so the synthetic orchestra-runtime session is no longer needed to hold the server open past its last agent pane. Nothing in the tree referenced that session.

[Service]
Type=simple
User=kami
ExecStart=/usr/bin/tmux -D -L orchestra
Restart=on-failure
RestartSec=1

Verified after install: MainPID=3073848, cgroup holds only the server, and the server stays alive with zero sessions.

deploy/orchestra-worker.service said User=orchestra while the installed unit ran as kami. Since the socket is /tmp/tmux-$UID, that mismatch would have pointed the worker at a different server. Corrected in fc4c29f.

F28 proof, live

Vehicle: disposable task 06G474FXXFW7V54RGF70ZS3SRG, forced onto workpc-claude, leased at 20:05:07Z under epoch 06G49MW2CHNVT01SHEKVYJ2170, pane orchestra-06g474fxxfw7v54rgf70zs3srg-59bf1ee9:1.0, pane pid 3012539.

Assertion	Result
Pane survives worker restart	3 restarts, pid 3012539 every time
Same agent identity	unchanged
Same lease epoch continues	06G49MW2CHNVT01SHEKVYJ2170
Worker reconciles lease and pane	leases and sessions rehydrated each time
No TaskReleased from a restart	event log empty after seq 385
No replacement TaskLeased	same
Subsequent renewal succeeds	00:25:11 local, v6 → v7, progress_sha recorded

The event log is the load-bearing evidence. The only TaskReleased on this task is seq 382 at 20:04:06Z, from an earlier failed launch, well before the first restart at 20:08:33Z. The restarts appended nothing at all.

Forcing a renewal by shortening the worker's cached until does not work. The worker re-hydrates the authoritative deadline from the coordinator on startup, before any renewal tick runs. The cache is not a lever.

Forcing a task onto one harness

config.jsonc gave workpc-claude the capabilities ["code", "review"], and workpc-opencode ["code", "review", "opencode"]. No capability selected claude uniquely, so an operator test could not pin placement. claude was added to workpc-claude, mirroring the convention opencode already followed. Router rejections at /v1/router/health named the gate directly, which is what made this a two-minute diagnosis instead of an afternoon.

F29: the browser could not act on a leased task

validateTransition fences TaskReleased, TaskBlocked and TaskCompleted on a leased task against the live harness_id and lease_epoch. The UI action handler sent neither, so all three returned 409 on exactly the tasks the UI listed them as enabled for.

Found by trying to park run 4: eight block attempts, stable v8, all 409 task version conflict. The workaround was to block during the queued window, where t.Lease is nil and the fence does not apply.

Fixed in efd0a5e. The fence exists to reject a stale writer, not the operator, so the handler now carries the lease it just read. The version CAS on the append still rejects a racing write. A missing request body also wrote into a nil map, which is fixed alongside.

F30: an unpushed release pinned the worker's only slot

A release transaction that never reached anchor_pushed has no artifact. tx.Ref is empty and no successor can pick anything up. The event handler kept its session mapping alive anyway, so once the pane was gone the mapping was immortal. health() reports ActiveTask straight out of w.sessions, so the coordinator saw the harness as permanently busy and never leased to it again.

Live on workpc-claude, stuck at phase prepared behind this error:

adapter: invalid handoff answer: invalid handoff authored field: prose smuggled into list

It produced no log line for fifteen minutes. Freeing it needed hand surgery on /var/lib/orchestra/worker-state/workpc-claude.json with the worker stopped.

Fixed in 2dc90bd. The mapping is kept only while an anchor actually exists, and the transaction is dropped with it, since nothing can advance it.

Worth recording separately: the rejected handoff artifact came from an agent running the operator's direct-prose stop hook. That is a live interaction between an operator hook and the handoff contract, not a defect in either.

Ledger

F6   closed
F14  closed
F15  closed by detection, transport fix pending live proof
F16  closed, both branches live-proven
F17  fixed, isolated live proof
F18  open observability, non-blocking, cost diagnosis time twice this session
F19  fixed
F20  fixed, live receipt still pending
F21  reading and artifact halves live-proven; accepted transition pending
F22  fixed, live proof pending
F23  closed as already implemented
F24  open correctness, dormant on current topology
F25  fixed, live-proven
F26  fixed, live-proven
F27  fixed, live-proven
F28  CLOSED, live-proven in both directions
F29  fixed in efd0a5e, needs a coordinator rebuild
F30  fixed in 2dc90bd, needs a worker rebuild
F31  fixed in 6523002, needs a worker rebuild, BLOCKS every rotation

Deployment state

Both fixes are committed and unbuilt. The running coordinator and worker are still fbaaf79. Run 5 should start from a deployment carrying efd0a5e and 2dc90bd, or it will hit F30 again the first time a handoff artifact is rejected.

F28 runtime-death direction, live

systemctl stop orchestra-tmux at 00:39, restarted at 00:39:37 as MainPID 3073848. Every pane died with the old server.

20:45:07Z  renewal window opens, no TaskLeaseRenewed appended   F16 refused
20:55:07Z  lease deadline
20:55:50Z  state queued v8, lifecycle reclaimed, failure_class lease_expired

Runtime death is now distinct from worker deployment. F16's missing-pane branch fires on real execution loss, which is what it was written for.

The same window reproduced F30 a second time, on the deployed fbaaf79 worker that lacks 2dc90bd: session still pinned, release transaction stuck at prepared, same rejected handoff artifact.

F31: every rotation failed on a length check that named the wrong cause

Two different tasks, two different agents, one error:

adapter: invalid handoff answer: invalid handoff authored field: prose smuggled into list

parseHandoffAnswer builds Action by joining the agent's NEXT and WHY answers with " — ". Validate then held that join to maxAuthoredLine, 200 characters. The prompt asks for a sentence each and names no budget, so two ordinary sentences do not fit.

The message named a branch the answer cannot reach. parseHandoffAnswer splits on newlines and trims every field, so no authored field ever contains "\n#". Length was the only reachable cause, and the agent was never told what to shorten. It could not self-correct, so the same text failed on every retry.

This is why run 4 and the F28 disposable task both died at rotation, and why each left the stuck transaction behind that pinned the worker slot.

Fixed in 6523002: Action carries the budget of both lines, the error names the actual length and limit, and the prompt states the limit.

F31 blocks run 5 outright. Run 5's chain is four rotations.

Deployment of 93338b7, and F32, 2026-08-28

Both halves deployed from one revision, built from a throwaway worktree because the shared checkout still carries 20 uncommitted paths from another session.

coordinator  93338b72cec5facdb991d5dfa56eb9d3195749d0  image sha256:8b57491da293  readyz 200
worker       93338b72cec5facdb991d5dfa56eb9d3195749d0  sha256 168bf20724ec7e03bb5bd19320d4b698d31441d15ce3706d0498d1c85ebd044f
tmux runtime MainPID 3073848, untouched across the worker install

The worker install is itself further F28 evidence. A real deployment left the runtime and its cgroup alone.

F29, live-proven

Blocking the F28 disposable task while leased returned HTTP 200 and moved it to blocked v10. The same call returned 409 eight times before this deploy.

F30, live-proven, and the defect hiding behind it

The stuck releases entry cleared on the block, which is the fix working. The session did not clear. It was marked quarantined instead, because Kill returned an error for a pane that no longer exists.

F32: an empty tmux server answers differently

tmux has-session -t '=<gone>'   →  "no current target"   exit 1

hasSession recognised can't find session, no server running, no sessions and a socket-missing case. It did not recognise no current target, so a missing session read as a real error.

This is a side effect of dea56e4. Before the runtime became its own unit, the server exited with its last session and answered no server running, which was already handled. tmux -D keeps the server alive past its last pane, so this reply had never been produced before today.

Consequences: Kill fails for an already-dead pane, the quarantine never clears, and the worker's only session slot stays pinned. AgentStatus shares the helper and errored where it should report exited, which means F16's missing-pane branch was refusing renewal by error rather than by detection.

Fixed in 92adf04.

Ledger

F28  closed, live-proven in both directions
F29  fixed in efd0a5e, DEPLOYED, live-proven
F30  fixed in 2dc90bd, DEPLOYED, partially live-proven, rest blocked by F32
F31  fixed in 6523002, DEPLOYED, awaiting run 5's first rotation
F32  fixed in 92adf04, needs a rebuild, BLOCKS leasing on workpc-claude

Run 5, first attempt: the launch never went off, 2026-08-28

Filed as kami/test-e2e#5, ingested as 06G4A4F0TFXKZHJE48N05XN1HG, placed on workpc-claude by normal routing. Issue filed to confirmed launch: 44 seconds.

Then nothing moved for seven minutes. The pane held the launch text unsent:

❯ @.orchestra/launch.md is your complete Orchestra launch instruction. Read
  .orchestra/launch.md now and follow it.

0 tokens        ⏱️ 0m 0s        session ... - none

F33: an editor that never rendered confirmed the submit

first_submit_at  21:13:19.570302125Z
confirmed_at     21:13:19.577191672Z
confirmation     editor_cleared   submit_attempts=1

Seven milliseconds. ConfirmInput's first poll ran before the TUI rendered the pasted text, found an empty editor, fell through to the idle branch, and called that proof of submission.

The two launches that worked earlier took submit_attempts=2 over roughly 500ms. The difference between a good launch and this one was scheduling luck, which is why it had never been seen before.

The cost is worse than a failed launch. The coordinator believes the agent is working, so the lease renews against a pane that will never produce anything, and the failure only surfaces at expiry half an hour later.

Fixed in 32220d9. An empty editor counts as proof only after it has held the text, or after it has stayed empty past a settle window. Text seen and then gone still confirms immediately, and so do busy, blocked and queued. With the fix the failing case reaches the existing resubmit path instead: the text appears, is recognised as unsubmitted, and Enter is resent.

Recovery, deliberately not by hand

Pressing Enter in the pane would fix this run and prove nothing. The lease is left to expire and relaunch on its own, which is the autonomous path run 3 already proved. The worker carrying 32220d9 is deployed first, so the relaunch uses the fixed confirmation.

Ledger

F29  DEPLOYED, live-proven
F30  DEPLOYED, live-proven
F31  DEPLOYED, still awaiting a first rotation
F32  DEPLOYED, live-proven, slot reaped and freed
F33  fixed in 32220d9, needs a rebuild, BLOCKS every launch nondeterministically

Run 5, second attempt: two more layers, 2026-08-28

The stuck launch was recovered by hand, one Enter into the pane. That is an operator recovery of a keystroke Orchestra believed it had already delivered, not a lifecycle intervention, and F35 turns it into an action for next time.

F34, live-proven while being written

01:33:37  first renewal allowed, no baseline, lease extends 21:43 -> 22:03

The pane had not changed since 01:13 and held unsent text. The renewal gate exempts a lease with no baseline, which grants a full free period to exactly the case the gate exists to catch. Fixed in 92f32d6 by baselining the progress hash at launch.

F35: nothing could re-poke a pane

The UI offered grant_approval, deny_approval, handoff, release, block and complete. Orchestra can put text in an editor and be wrong about whether it landed, and the only recovery was to destroy the lease and wait out expiry.

resubmit presses Enter on text Orchestra itself submitted. No lifecycle change, so no event; fenced like an approval, on a live worker-owned pane at the capture revision the operator was looking at. In 92f32d6.

The gate, half cleared

01:36:35  phase request accepted: frame to research
01:36:35  acceptance delivered, confirmation=queued
01:36:35  phase changed: session rotating

Further than any previous run. work_phase reached research and the request artifact was consumed.

F36: the two halves of the handoff contract disagreed

before  adapter: invalid handoff answer: invalid handoff authored field: prose smuggled into list
after   adapter: upload handoff: invalid handoff meta

F31 cleared the parse, and the release then failed one stage later. handoffReason has emitted phase_changed since phase rotations landed. The validator's reasons list was never extended, so every phase rotation built a handoff it then refused.

Fixed in b9ca365. The test asserts the property, not the constant: every reason the adapter can produce must survive Validate, including its fallback.

Pattern worth naming

Six defects tonight, each hidden behind the one before it. F30 was reachable only once F29 let an operator act, F32 only once F30 tried to reap, F34 and F36 only once F33 and F31 let execution get that far. Tests passed throughout. The only thing that surfaced any of them was running the system.

F37: the handoff carried the write Orchestra asked for

One stage past F36:

adapter: upload handoff: invalid handoff command: must not point to a handoff or report

Handoff.Command is the last command observed in the pane. The handoff prompt tells the agent to write .orchestra-handoff-report.md and stop, so that write is almost always the last command there. .orchestra-handoff-report.md matches every alternative in circularCommand, so every rotation failed on Orchestra's own instruction.

Fixed in cb7782d. lastObservedCommand skips commands the validator would call circular. The rule is asked for through continuity.IsCircularCommand rather than restated, because restating a rule in two places is exactly what produced F36.

Run 5: the continuity chain, live and autonomous, 2026-08-28 01:47

The first successful phase rotation in this project's burn-in history.

handoff_ref           abe51f5c663f4d59bba2c98c0418554278ec2a9b75bfc504bd36757666e3948a
transaction_id        06G4A9VSBE5N5BF8BE3J0PSAER    the one that had stuck at "prepared"
anchor_sha            c643aaab87b4b1f79b9f1d909bf8788a0e25fe61
pickup_acknowledged   true
new epoch             06G4AC5KX225JGXAEWH52NNPB0
new pane              created 01:47:09
session phase         research
task                  leased v9, work_phase research, lifecycle started

Step	Evidence
Handoff answer parses	F31, no invalid handoff answer
Handoff validates	F36 and F37, handoff_ref exists
Anchor pushed	anchor_sha c643aaab
Release commits	transaction gone from releases
Predecessor slot frees	old session dropped
Successor leases and starts	new epoch, new pane, phase research
Pickup validates	pickup_acknowledged: true

No lifecycle shepherding. The only operator action in this run was one Enter into a pane holding unsent input, which F35 turns into an action.

Two fixes proved themselves incidentally in the same event. The relaunch logged submit_attempts=2, so F33's resend path does real recovery rather than existing only in tests. The new lease carries progress_sha at once, so F34's launch baseline survives into the real worker path.

Remaining gates

research  -> research.json sealed -> request -> rotation -> plan launch carries accepted research
plan live -> correction comment -> HumanDecisionRecorded -> RemoteTurn -> DecisionNotice receipt, once
plan.json sealed -> request -> rotation
implement launch -> correction above accepted plan above accepted research

The correction is held until the plan successor is definitely live. Posting earlier still exercises F20 and F23, but it cannot prove a mid-plan correction outranks accepted research and the current trajectory.

F38 and F39: the agent guessed a schema, and was never told it guessed wrong

Run 5 sealed research.json at 01:47:52 and wrote {"from": "research", "to": "plan"}. Nothing moved for five minutes. The operator noticed before the system reported anything.

research artifact: json: cannot unmarshal string into Go struct
field Research.dead_ends of type workphase.DeadEnd

F38: the brief said Seal .orchestra/research.json before you ask and described the contents in prose. It never stated the schema. The agent wrote dead_ends as strings where the decoder wants {tried, why_failed} objects. Everything else in the artifact was correct.

F39: requestPhase refused through recordError alone. answerRefusedPhase ran only on a coordinator 409, so a local decode failure reached nobody. The session parked at a boundary with no feedback, and worker health holds one error slot (F18), so even that trace was overwritable.

F39 is the serious half. F38 alone costs one corrected file. F39 turns any local refusal into a silent stall.

Fixed in 0bd86e2. The brief carries the shape, and every local refusal now reaches the agent. TestPhaseSealSchemasDecode decodes each documented shape with the function the worker uses: flipping dead_ends back to strings reproduces run 5's exact error at build time.

Run 5: the whole chain, live, 2026-08-28 02:05

Filed as kami/test-e2e#5, task 06G4A4F0TFXKZHJE48N05XN1HG, harness workpc-claude, both halves on cd1675a.

01:13:19  launched, then stalled: F33 confirmed an editor that had not rendered
02:00:05  worker restarted on cd1675a
02:00:06  phase request refused, delivered into the pane (F39)
02:02     agent rewrote research.json, unprompted by any human
02:03:00  research to plan accepted
02:03:31  plan session launched carrying accepted research
02:04:05  plan.json refused: changes[0].intent 840 characters exceeds the 500 bound
02:04:07  operator correction posted to the issue
02:04:10  delivered to the live plan session, decision 06G4AG40NR866DVD875ERFZT70
02:04:30  plan to implement accepted
02:04:37  plan.json names it: "Human decision: JSON keys must be snake_case..."
02:05:06  implement session launched

Three seconds from comment to delivery. delivered_decisions holds exactly one id, so the notice did not repeat.

The authority order, as rendered

## Goal
## Acceptance
## Current human decisions
   - correction (operator_instruction): for --json, use snake_case keys.
     keep the default text output byte-for-byte unchanged.
## Current phase              implement
## Repository rules
## Verified git state
## Accepted research
## Accepted plan
## Continuity from the previous session
### Dead ends already tried

Human correction above accepted plan, above accepted research, stale continuity below all of them. That is the target chain, live.

What each fix bought, observed rather than argued

• F33: the 02:03:31 and 02:05:06 launches both logged submit_attempts=2. The resend path is doing real recovery on every rotation.
• F39: two refusals delivered, one for research.json and one for plan.json. The agent corrected both without a human touching anything. Before tonight each would have been a silent stall until lease expiry.
• F38: the plan brief carried the plan schema, so the plan artifact failed on a length bound rather than on a guessed shape.
• F20 and F23: one comment, one decision, one delivery, one receipt.

Standing caveat

Run 5 is implementing, not finished. The chain this run set out to prove is proven. Completion, the review phase, and the final report are still ahead.

F40: the terminal phase never learned how to finish

Run 5's review agent finished at 02:12 and stopped. No phase-request.json, no handoff, no .orchestra/done, and nothing in the worker journal after 02:06:21.

legalPhaseTransitions makes review terminal: its only move is backwards to implement, for rework. A review that passes has nothing to ask for.

The worker has always finalised on .orchestra/done (cmd/orchestra-worker/main.go:546). No brief ever named that file. Grepping a rendered launch.md for it returned zero matches, in every phase.

So the completion signal existed on one side of the contract only. An agent whose review passed had no instruction at all, and stopping was the correct reading of what it had been told.

Fixed in the phase brief: the terminal phase now states the marker, when to write it, and that it is exclusive with asking to go back. TestTerminalPhaseNamesTheCompletionSignal also asserts the negative, so a phase that can still ask is never told to finish instead.

F41: harness chrome counted as agent progress

Run 5's review agent produced nothing after 02:12. The 02:46 renewal was granted anyway, and the lease moved to 03:16. Each renewal window bought another 30 minutes, so recovery by expiry had no bound.

Reproduced the worker's stored progress_sha byte for byte from the live pane, so the branch taken at cmd/orchestra-worker/main.go:1020 was progress != l.ProgressSHA. Not IsBusy: the 200-line capture held zero esc to interrupt markers, so AgentStatus returned idle. Not an empty baseline either: it held 3d93ca89.

PaneProgress dropped prompt lines only, so Claude Code's status footer stayed in the digest. The mutable fields there are the rolling usage percentage, the context counter, and the version notice. A 180-second sample showed no churn, so this leaked intermittently rather than constantly. One tick inside a window was enough.

Fixed in 2417a39. PaneProgress cuts from the editor's lower rule and then trims the spinner summary and version notice above it. AgentStatus still reads the raw capture, so the busy markers living in the footer are unaffected.

Proven live in both directions on the first window after deployment: the 03:13:48 renewal took version 28, the 03:16:13 window was refused, the lease expired at 03:18:43, and review relaunched at 03:20:14 with the F40 brief. The relaunched pane then showed a new footer field, 5h: 15%, which the old digest would have counted as work.

Lease TTL moved to 5 minutes in the same commit, from domain.LeaseTTL, with renewal at half of it. Reclaiming a stalled pane happens only at expiry.

F42: the result commit refused the completion marker

The relaunched review agent wrote .orchestra/done at 03:20:48, 34 seconds after launch. The worker recognised it, confirmed the agent idle, and then failed the result commit once every five seconds:

stage result: The following paths are ignored by one of your .gitignore files:
.orchestra/done

internal/herdr/adapter.go:132 writes .orchestra/.gitignore containing *, so the marker is ignored. The staging step passed :!.orchestra/done, and git refuses an add whose pathspec names an ignored path. Reproduced in a scratch repo on git 2.55.0, both the failing and the working form.

Fixed in dcd9af4: the exclusion names the directory, :!.orchestra. That holds whether or not the inner .gitignore exists.

The failure retried for 22 minutes with the task stuck in review and nothing observable outside the journal, because each identical error overwrote the single last_error slot. F18, again, and the second time in one night that it turned a five-second loop into archaeology.

Run 5 reached TaskCompleted

On the first tick after the dcd9af4 restart at 06:47:50:

• task state completed, lease released, .orchestra/done removed
• pane closed, no orchestra-* sessions left on the tmux socket
• result branch pushed: refs/heads/orchestra/scratch/orchestra-06g4a4f0tfxkzhje48n05xn1hg-73a2e4fd at fb15c61

Two links of the tail did not run, and neither is a defect. The deployed test-e2e project declares only id, machine_affinity, repo and worktree_root, so no quality gate exists to run. The deployed config.jsonc has no sources and no delivery keys, so no PR could be opened.

Proven chain: review, done recognised, finalise, result branch pushed, TaskCompleted. Submission, human PR review and merge remain unproven, and they need registry configuration rather than code.

Run 6, 2026-08-28: the PR tail, and the baseline conformance run

The last handoff said submission, human review and merge needed configuration rather than code. That was wrong. The whole event log held zero ReviewRecorded and zero TaskSubmitted: nothing in the running system ever called either endpoint, so the publisher, the merge reflection and the human trust boundary had no entry point at all. Four links were missing, and three more defects surfaced while proving them.

F43, the reviewed change had no path to the human

The worker's completion tail called federation.Client.Complete directly and emitted TaskCompleted. POST /v1/tasks/{id}/review and POST /v1/tasks/{id}/submission existed and had no live caller; task pr is named in six documents and implemented nowhere. GiteaPublisher.Push also ran git push from projectRoots[project]/<task> on the coordinator, a directory that does not exist for a worker-owned project whose worktree is on workpc.

Fixed in e8d04d7:

• finalize commits first and runs the gate against the committed tree, so GateResult.SHA is the commit being submitted. CheckSubmission requires gate sha, review sha and head sha to be one commit, which a gate run on the pre-commit tree can never satisfy.
• POST /v1/federation/workers/<id>/submit seals the review and submits. A blocking review returns the task to implementation; a project with no forge still completes directly.
• The reviewer's brief names .orchestra/review.json and no longer claims a diff is supplied. Nothing populates agentctx.Evidence on the federated path, so the brief described material the session never received.
• Push asks the forge what the branch holds before reaching for a checkout.

F44, the Gitea source ingested Orchestra's own pull requests

The first submission this deployment ever made, kami/test-e2e#8, came back one minute later as task 06G4E83E4KRXM8DS90M2648MGM with the submission packet as its description. That task would have implemented, reviewed and submitted again, opening a pull request per cycle. The issues endpoint returns pull requests and nothing filtered them. The webhook had the same hole from the other side: a pull_request delivery leaves the issue key empty and would have appended a task numbered 0 with no title.

Contained by blocking the spawned task, fixed in 6ccc755.

F45, the submission reflection loop was dead code

if len(pullRequests) > 0 guarded the loop 600 lines before the Gitea wiring that writes to that map. The length was always zero, the goroutine never started, and a merged pull request could never complete its task. Live: a trusted comment on PR #8 moved nothing until 4af9880 moved the block below the wiring.

F46, stale review findings survive a changes-requested round trip

The worktree is not recreated, so .orchestra/review.json from the first review is still present when the second one starts. A reviewer that writes .orchestra/done without rewriting it would have the earlier findings sealed against the new commit, and a stale pass is indistinguishable from a fresh one. Observed at 10:57:48 with a file from 10:55:45. That reviewer did rewrite it, so the run stands. Fixed in 063a3ab.

Baseline conformance run

Task 06G4E6F69AKP2PA00S28D7ASBC, issue kami/test-e2e#7, workpc-claude, tmux backend. Coordinator 4af9880, worker e8d04d7. No manual lifecycle intervention: every event below was emitted by the plane.

06:41:14  TaskCreated          issue #7 polled
06:41:37  WorkPhaseChanged     frame -> research
06:42:57  WorkPhaseChanged     research -> plan
06:45:12  WorkPhaseChanged     plan -> implement
06:46:27  WorkPhaseChanged     implement -> review
06:47:48  ReviewRecorded       blocking 0, sha 41dee802
06:47:49  TaskSubmitted        PR 8 opened, sha 41dee802
06:55:11  TaskChangesRequested trusted comment by kami
06:55:11  WorkPhaseChanged     review -> implement
06:57:47  WorkPhaseChanged     implement -> review
07:00:09  ReviewRecorded       blocking 0, sha 0d1585df
07:00:09  TaskSubmitted        PR 8 reused, sha 0d1585df
07:02:07  merge                by kami
07:02:11  TaskCompleted        receipt binds merge f48e363d

Acceptance, each checked against the artifact rather than the log line:

• The gate ran against the exact submitted commit. Gate artifact bd3549d6: {"command":"bash -n scripts/*.sh && bash scripts/orchestra_e2e_healthcheck.sh","exit_code":0,"sha":"0d1585df"}.
• TaskSubmitted binds all five: gate sha, review sha, result sha, remote ref origin/orchestra/06G4E6F69AKP2PA00S28D7ASBC, and PR 8. The PR head reads 0d1585df at the forge.
• Resubmission reused PR 8 with a new gate ref and a new review ref.
• A trusted human reopened the task and the change reached merged master: --quiet is documented in --help, which is what the comment asked for.
• Merge produced the completion receipt: merge_sha f48e363d, submitted_sha 0d1585df, submission_ref 06G4EASR63SZ1WWRYHR62WJQS4.
• Final state completed.

Freeze this as the baseline. One caveat on the reopen: Gitea returns 422 for a review on your own pull request, and Orchestra opens the PR as the same forge user the operator reviews as. REQUEST_CHANGES is therefore unreachable in this deployment and the comment path is what was exercised. A separate bot account for ORCHESTRA_GITEA_TOKEN would restore it.

Configuration this needed, for the record: quality_gate on test-e2e in both config.jsonc (coordinator) and /etc/orchestra/worker-projects.json (worker, which is the copy finalize actually reads), plus ORCHESTRA_REVIEW_ACTORS=kami in the compose .env.

Plan machinery, written 2026-08-28, unproven live

Four commits above orchestra-conformance-v1. Nothing deployed. Full detail is in HANDOFF-2026-08-28-plan-machinery.md; this is the ledger entry.

Commit	What
822f086	Research findings gain id and confidence. The advertised schema becomes true.
57c028f	plan.md replaces the four-bullet-list plan artifact. Renders verbatim into the implement launch.
a221502	Plan phase progress Orchestra establishes, plus the project verification allowlist.
c76112a	Typed plan mismatch and the Orchestra-owned reopen.

The burn-in it needs

Configuration first: test-e2e needs a verification block in the coordinator's config.jsonc. Absent policy refuses every plan command, so without it a plan seals and no phase can ever verify. Both halves must be rebuilt and redeployed together, because the worker calls two new federation routes.

Then the ladder:

1. plan seals three detailed phases
   -> grep the rendered launch.md for the phase-three verification command
2. phase 1 requests verification
   -> worker runs the exact plan command
   -> durable state says verified
3. rotation
   -> successor gets the complete original plan
   -> phase 1 verified, phase 2 explicitly current
4. human correction lands above the plan
5. phase 2 contradiction
   -> PlanMismatchRecorded, no improvisation
   -> old plan retained until the revised one seals
   -> fresh launch carries the revised plan in full
6. a plan command outside project policy
   -> refused at seal time, on the planner
7. a legacy plan
   -> "phase progress unavailable", task still completes

Steps 1 and 3 are the rungs that matter for smaller local models. Steps 6 and 7 were added from what this session's implementation surfaced.

Freeze as orchestra-plan-v1 if it passes.

F18 is now blocking

Three more instances landed with this change: every refusal in the new plan machinery records into the same single last_error slot. That is the fourth, fifth and sixth instance. The plan burn-in will produce exactly the five-second retry loops the bounded observation ring exists to make visible, so fix F18 before running it.