orchestra/BURNIN.md

Burn-in: live conformance, not more architecture

Written 2026-08-26. The workflow is feature-complete enough to exercise. What remains is empirical: run the same real task shape through each harness and classify what breaks. Isolated tests are insufficient by design here, and only the live owner path establishes conformance.

Do not add workflow features while this is running. Findings decide the next implementation work.

Burn-in build identity

77a2b323fabcf080d7542061ae2d7b3c34eef5b7

Both halves must report exactly this revision before a task is created. Neither needs a credential now: the coordinator prints it in docker logs orchestra-api and the worker in journalctl -u orchestra-worker. The same object is at GET /v1/admin/diagnostics and GET /v1/federation/workers behind the operator login.

Build both with deploy/build.sh <outdir>, which refuses a dirty tree. Later documentation-only commits do not change this identity, so a rebuild either passes this revision explicitly or accepts the new one and redeploys both halves. Never one half.

Deployment state, 2026-08-26 18:35

Half	State
Coordinator (homesrv)	Deployed at 6f9300b. Rebuilt with --build-arg BUILD_REVISION, recreated, /readyz ready, self-reporting the revision in its log
Worker (workpc)	Staged, not installed. ~/orchestra-deploy/orchestra-worker sha256 2b1c43071eaf6b5c15b110de39f204038a9629897e0ce3dacf45be96a6e6529e. Installed binary is still the 2026-07-30 build

Two operator steps remain, both needing root:

sudo install -m 0755 /home/kami/orchestra-deploy/orchestra-worker /usr/local/bin/orchestra-worker
sudo systemctl restart orchestra-worker
journalctl -u orchestra-worker -n 5 --no-pager   # must print revision 6f9300b...

A third blocker, found while preparing the pane check:

herdr is not running on workpc. herdr status server reports not running, the socket at /home/kami/.config/herdr/herdr.sock refuses connections, and its log stops at 2026-07-30. workpc-opencode therefore cannot start a pane, even with the worker installed. The worker logs serving harness workpc-opencode (opencode) on herdr backend at startup without touching the socket, so that line is not evidence the backend is reachable. herdr is an interactive terminal workspace manager: running herdr in a terminal launches or attaches to the persistent session and starts the server. Confirm with herdr status server before creating a task.

Then scrub the pane environment before letting an agent run.

Where the pane environment actually comes from

The two workpc harnesses inherit different environments, so one scrub does not cover both.

• workpc-opencode (herdr backend). The pane is created by the herdr daemon, which is a separate long-running process. It inherits herdr's environment, not the worker's. Scrubbing worker.env does nothing here. Whatever environment herdr is started with is what every opencode pane gets.
• workpc-claude (tmux backend). TmuxBackend.StartAgent runs tmux new-session through exec.CommandContext with no Env set (internal/herdr/tmux.go). If the tmux server is not already up, the worker starts it, and that server inherits the worker's full environment. Every claude pane then inherits it too.

This exposes a conflict the scrub alone cannot resolve. The worker legitimately needs ORCHESTRA_WORKER_TOKEN (or ORCHESTRA_WORKER_TOKEN_<ID>) and ORCHESTRA_FEDERATION_ADMIT_TOKEN, and deleting them breaks the worker. Keeping them means the tmux path hands them to the agent. Closing it needs either a filtered cmd.Env when the backend spawns a pane, or a tmux server started separately with a clean environment. Not on flow 1's path, because flow 1 is opencode only. Close it before step 6.

Baseline: one harness, one path

workpc-opencode only. Codex is a coverage gap, not a prerequisite, and no worker runs on homesrv. Prove one real path first:

real vikunja task
→ workpc-opencode
→ frame/research/plan
→ trajectory gate if configured
→ implement
→ independent review
→ task pr
→ human merge
→ completed

Then raise difficulty in this order:

1. opencode boring success
2. opencode mid-session human correction
3. opencode forced rotation
4. opencode pr rejection -> fix -> merge
5. opencode reconcile outage/recovery
6. representative cases on claude
7. add the codex adapter/config
8. cross-harness rotation: opencode -> claude/codex

Herdr transports, so nobody misdiagnoses again

No entry in the live config.jsonc sets backend or address, so all six resolve to <machine>:9245 (registry.defaultHerdrPort). Both ports are closed. That says nothing about the workpc harnesses: they are worker-owned, and the coordinator logs worker-owned on workpc; coordinator probe skipped for each. The worker reaches workpc-claude over a tmux socket and workpc-opencode over /home/kami/.config/herdr/herdr.sock. Do not diagnose harness availability from a TCP probe of 9245.

Evidence to record per run

One row per run. The launch instruction is now dumped to <worktree>/.orchestra/launch.md (herdr.LaunchContextFile) at every launch, local and federated, so the context is auditable without reading pane scrollback.

harness            (claude | codex | opencode)
task id
lease epoch        (every epoch, if the task rotates)
session ids        (pane ids per epoch)
human decision ids
launch context     (sha256 of .orchestra/launch.md, per launch)
handoff ref
git sha            (at every boundary: launch, each handoff, review, submit, merge)
review ref
submission ref
pr id
completion receipt

The five flows

1. Boring success. task → research → plan → implement → review → pr → merge.
2. Mid-session correction. Agent is doing A, the human says B, the next verified boundary delivers B, and A becomes history rather than a competing instruction.
3. Rotation. Agent A hands off, agent B resumes. Switch harnesses across the boundary if both are up.
4. Human rejection. Submit sha A, comment on the pull request, task reopens, fix at sha B, a fresh review, the same pull request, merge.
5. Failure path. Human source goes down, the reconcile streak escalates, the session hands off, the successor lease is refused, the source recovers, and a corrected successor starts.

What to inspect by hand

For every launch, read .orchestra/launch.md and ask only:

does this agent know
- what the task actually wants?
- what was most recently decided?
- what phase it is in?
- what earlier material is merely historical?
- what to do next?

Then compare that against what the model did.

Classify before fixing

Every failure gets exactly one label before any code is written:

authority bug            the wrong thing outranked the right thing
context-selection bug    the renderer showed or hid the wrong material
lifecycle bug            state, lease, phase or event handling is wrong
adapter/harness bug       the pane, occupancy, boundary or prompt path is wrong
model-following failure  the context was right and the model ignored it
operator-policy gap      credentials, deployment or configuration

This matters because the cheap response to every failure is another prompt rule, and prompt rules accumulated to compensate for lifecycle or adapter bugs are how the enforcement boundary rots. A model-following failure is the only class a prompt change should ever answer.

Pane credential cleanup (operator, in parallel)

The agent pane inherits the worker's environment. On workpc the worker is orchestra-worker.service, User=kami, EnvironmentFile=/etc/orchestra/worker.env (mode 0600, root-owned, deliberately unreadable here). Remove from that file, and from anything else the pane inherits:

gitea mutation token
vikunja mutation credentials
orchestra operator / system token

Retain only ORCHESTRA_AGENT_TOKEN and credentials a specific task genuinely needs. If an agent needs a forge operation, it asks Orchestra to perform it.

Then verify from inside a real pane:

git push                 # must fail unless Orchestra supplied the auth
curl .../v1/tasks/<id>/submission   # must be 403 for the agent surface
tea pr create            # must have no usable mutation credential

Note what is not enforced by code: nothing in this repo scrubs the pane environment, and a fake git earlier in $PATH is not enforcement because /usr/bin/git bypasses it. Credential isolation is the enforcement. Confinement of network, filesystem and destructive commands belongs to whatever launches the process (herdr, a container, systemd, bubblewrap), not to Orchestra, which supplies identity and policy inputs.

Exit criterion

Roughly 10 to 20 real tasks, with the five flows covered on each harness that is actually up. Then read the classified failures and decide the next implementation unit from them.

Live findings, 2026-08-26 21:35

Both halves are paired at 6f9300b, and herdr is up (0.7.5, protocol 17). What remains before flow 1, and what the in-pane credential checks will really show.

Flow 1 has no repository to run in

workpc-opencode declares exactly one project, test-e2e, pointing at /tmp/test-e2e and /tmp/test-e2e-worktrees. Neither path exists on workpc after today's reboot, and /var/lib/orchestra/repos does not exist there at all. Nothing can be leased to this harness until it has a project backed by a durable local repository.

The only configured task source is Gitea kami/correx at https://gitea.kvmx.ru (ORCHESTRA_GITEA_URL/OWNER/REPO), so a real task means a correx issue. correx already has machine_affinity: ["homesrv","workpc"] in the coordinator's config.jsonc. What is missing is a workpc-local entry:

// /etc/orchestra/worker-projects.json, on workpc
{
  "correx": {
    "repo": "/home/kami/orchestra/repos/correx.git",
    "worktree_root": "/home/kami/orchestra/worktrees/correx",
    "remote": "origin"
  }
}

Not /tmp. The burn-in outlives a reboot.

There is no vikunja ingest

This repo has no Vikunja provider. /readyz reports gitea and jsonl. A "real vikunja task" cannot enter Orchestra today: it arrives as a Gitea issue or through the JSONL watcher.

git push from a pane will succeed, and no Orchestra setting stops it

Corrected 21:45. The mechanism is SSH, not HTTPS. HTTPS has no stored credential on workpc, where git push over https:// fails with could not read Username. Pushing works over ssh://git@gitea.kvmx.ru:2222 using kami's default RSA identity, which Gitea lists as the key named workpc. It carries no passphrase, so no agent is needed.

That identity is what lets the worker push, and panes run as the same user with the same home directory, so an agent inherits it. worker.env is irrelevant to this. Real isolation needs panes under a different unix user. Expect git push --dry-run to succeed from a pane, and record it as an operator-policy gap rather than a code defect.

The worker inherits ambient Git credentials by design: git() in cmd/orchestra-worker/main.go runs exec.CommandContext with no environment of its own. There is no separate credential path for the worker to hold something the pane does not.

The agent surface is currently unauthenticated

ORCHESTRA_AGENT_TOKEN is unset on the coordinator, and an unset surface token means the middleware performs no check for that surface. Probed live:

POST /v1/tasks/<id>/decision-request  -H 'X-Orchestra-Surface: agent'  -> 404 (reached the handler)
POST /v1/tasks/<id>/phase             -H 'X-Orchestra-Surface: agent'  -> 403 (refused)

The capability boundary holds. Authentication does not. Set ORCHESTRA_AGENT_TOKEN in the coordinator .env before the burn-in, and note the same is true of ORCHESTRA_MCP_TOKEN and ORCHESTRA_MAVEN_TOKEN, both unset.

Two more unset settings that change burn-in behaviour

• ORCHESTRA_FEDERATION_ADMIT_TOKEN is unset, so any caller may register a new worker identity. Existing identities stay protected, because registering an existing id with a different token is refused.
• ORCHESTRA_REVIEW_ACTORS is unset, which human.Trust reads as "anyone not explicitly ignored". Flow 4 will accept a task-moving comment from any Gitea actor. Set it to kami.

Flow 1 setup, 2026-08-26 21:45

Target: kami/test-e2e on Gitea, a throwaway repo, rather than correx.

Done:

• Ingest switched. ORCHESTRA_GITEA_REPO=test-e2e in the coordinator .env (previous file kept as .env.pre-burnin-20260826), container recreated, still reporting 6f9300b. The project id must equal the repo name, because main.go sets Project: os.Getenv("ORCHESTRA_GITEA_REPO"). test-e2e already exists in config.jsonc with machine_affinity: ["workpc"].
• Durable repo on workpc. Bare clone at /home/kami/orchestra/repos/test-e2e.git, worktree root /home/kami/orchestra/worktrees/test-e2e. Origin points at ssh://git@gitea.kvmx.ru:2222/kami/test-e2e.git, and git push --dry-run reports Everything up-to-date. The old /tmp paths did not survive the reboot and must not come back.

Remaining, needs root:

// /etc/orchestra/worker-projects.json
{
  "test-e2e": {
    "repo": "/home/kami/orchestra/repos/test-e2e.git",
    "worktree_root": "/home/kami/orchestra/worktrees/test-e2e",
    "remote": "origin"
  }
}

Then sudo systemctl restart orchestra-worker.

Store state, checked before the first run: 30 tasks, all test-e2e, with 23 blocked, 6 completed and 1 failed. All are July leftovers and none holds a lease. The coordinator has run ResumeAnsweredBlockers every second for hours without resuming any of them, so they are inert rather than merely quiet. The July stuck task 06FT6CKD9Y98AZRX6X8K3QXFZG is now failed.

A stale branch orchestra/scratch/oc-06ftgkjadcd2hwjn2zwjen90q4 exists on the remote from an earlier run. Harmless, but it is not from this burn-in.

Run 1: task 06G3YR34117MAYT6KEAC9RJHD0, 2026-08-26

Evidence ledger:

harness         workpc-opencode (herdr backend)
task            06G3YR34117MAYT6KEAC9RJHD0
source          gitea:test-e2e/1  "Add a --version flag to the healthcheck script"
attempts        3 leases: two nacked, third launched
pane            wN:p1, agent oc-06g3yr34117mayt6keac9rjhd0
agent session   ses_fc096de31ffelZeMngQEyBTxfh
worktree        /tmp/test-e2e-worktrees/06G3YR34117MAYT6KEAC9RJHD0
branch          orchestra/06G3YR34117MAYT6KEAC9RJHD0
head at launch  5e6c4783d4213a934dc486160b888b6014bc2d03
launch context  .orchestra/launch.md, 2138 bytes
phase           frame
decisions       none

What worked

• The launch context is right. Authority order, the ambiguity ladder, the phase brief with frame: ... Do not change code, Current human decisions: None recorded, and a verified git state with worktree, branch and head.
• The agent respected the phase. agent_status: done, no code changes, only Orchestra's own scratch directory in the tree.
• A bad launch nacked cleanly. The first attempt failed on the base checkout and the task went back to queued with lifecycle_phase: launch_nacked and attempt: 1. No orphan pane, no stuck lease.

Fixed during the run

• F1, authority bug, 2753a8d. Every federated launch died with effective intent: federation: 401 Unauthorized: unauthorized surface. GET /v1/tasks/<id>/intent exists for workers, and the authz worker-path exemption never included it. Twenty test packages passed throughout.
• F2, authority bug, 09e572f. The rendered goal was the issue title alone and acceptance read Not stated., because Gitea.event parsed the issue body and dropped it from the TaskCreated payload. The store already read description. Every Gitea-sourced task so far ran on its title.
• F3, self-inflicted, a0209a2. The launch dump left .orchestra/ untracked in a worktree whose own context said uncommitted changes: false. It would have polluted the gate, the review diff and the agent's git status.

Open, classified

• F4, adapter/harness. rotation ... activity degraded: activity unknown: adapter: resolve session file: adapter: harness "opencode" has no session-file resolver. Thrash and activity triggers are permanently degraded on opencode. Observable in the worker's last_error, non-fatal.
• F5, lifecycle. The router never leased this task. Every gate checks out by hand: worker online: true, herdr_status: reachable, fresh checked_at, supported_projects: ["test-e2e"], quota empty, no active leases, capability empty. A direct POST /v1/tasks/<id>/lease succeeded instantly. The refusal is upstream of Store.Lease and silent by construction (internal/router/router.go:197). All three runs were leased by hand.
• F6, observability. Worker health reports active_task: null while the store shows the task leased and herdr shows a live opencode agent in the task's worktree.
• F7, operator policy. ORCHESTRA_TUI_TOKEN is unset, and an unset surface token means no authentication for that surface. That is how the manual leases above were issued, unauthenticated, from another machine. The TUI surface is FullControl, so this is the whole control plane, not just the two agent request endpoints. Set the token.
• F8, correctness. human.Reconciler.Reconcile iterates every configured source for every task, so the three queued correx tasks have their external ids looked up in kami/test-e2e. A source should only reconcile the tasks that came from it. Likely why those three never lease.

Fix pass before run 2, 2026-08-26 23:50

Burn-in identity is now 77a2b32. The coordinator is deployed at it. The worker is staged at it, sha256 2a850f2d1102d390..., still needing root to install.

Closed

• F7, security. authz.RequireCredentials refuses startup when a full-control surface has no token, rather than logging it. ORCHESTRA_TUI_TOKEN is set in the coordinator .env. Verified live: an unauthenticated POST on the TUI surface now returns 401. Web is exempt because Sessions makes its login mandatory. ORCHESTRA_MCP_TOKEN, ORCHESTRA_MAVEN_TOKEN and ORCHESTRA_AGENT_TOKEN remain unset, so those surfaces are still unauthenticated for reads and for their three request endpoints. Bounded by capability, worth closing, not startup-fatal.
• F5, lifecycle. Every eligibility gate now records a router.Rejection, exposed at GET /v1/router/health and reset per pass. No gate was weakened. The live output immediately explained the three stuck correx tasks: worker has not declared project correx. A queued task in retry backoff was skipped before the candidate loop and recorded nothing at all, which is the shape that hid the original case; it now reports retry backoff until <time>.
• F8, correctness. Reconciliation is bound to task.Source, the provider:project identity the ingest stamped. A source that cannot prove it owns the task is skipped, and a task with no matching source reconciles to nothing and still launches. The integration fixture had encoded the bug: it ingested from jsonl and reconciled from gitea.
• F11, outage. The API entered a restart loop exiting with invalid event: until_ns required. ValidateEvent compared until_ns against time.Now() for TaskLeased and TaskLeaseRenewed, so a lease event that was valid when written failed validation once it expired. store.Open replays the tail after the snapshot and log.Fatals on the first invalid event, so the coordinator refused its own history. Validation of a durable event is now time-independent. Latent since the field existed: it needed a renewal in the post-snapshot tail plus a restart after that renewal expired.

Also found

• F9, lifecycle. An operator cannot release a leased task through POST /v1/tasks/<id>/release without knowing its harness_id and lease_epoch, because Store.Append fences lifecycle events on a leased task and the endpoint passes the request body through unchanged. Correct that only the owner releases, but there is no operator escape hatch.
• F10, hygiene. A coordinator-side release under a live worker leaves the worker renewing a lease it no longer holds. Combined with F11 that produced a per-second invalid-event log line.

Shared checkout

Another session is working on the auth and frontend layers in this same checkout. Two consequences, both handled: deploy/build.sh and the container image now build in a detached worktree of the revision they stamp, so no uncommitted work is compiled into a stamped binary and no in-progress edit blocks a deploy. Commits from this session name their paths rather than using git add -A. The first commit, 7f12c7f, predates that discipline and swept in whatever was uncommitted at the time, including web/src and internal/authn.

Run 1 is closed as a diagnostic

Task 06G3YR34117MAYT6KEAC9RJHD0 was released with reason abandoned diagnostic run, supplying the lease fence by hand per F9. It sits at attempt: 3 and the router will fail it on its next pass. It is not a conformance run: it needed three manual leases and its authority was built before the issue-body fix.

Run 2 preconditions

worker installed at 77a2b32 and restarted
coordinator at 77a2b32                     done
F7 closed                                  done
F5 exposed                                 done
F8 fixed                                   done
fresh issue, no manual leasing

Runs 2 and 3, 2026-08-27

Written from live evidence. Neither run was a conformance pass, and both were worth more than one: run 2 found three blocking bugs behind each other, and run 3 found two more plus the reason six isolated probes disagreed with production.

AUDIT.md is uncommitted and owned by another session, so the burn-in ledger lives here.

Fixed, with the revision each landed in

• F12, lifecycle, 0d67af9. Store.QuotaSince reported an empty window as unknown, QuotaAvailability fails closed on unknown, and every herdr in config.jsonc declares a quota limit. The only producer of a receipt is a completed lease, so nothing could ever be leased. An empty window is now observable zero. A receipt that declares its own consumption unknown still fails closed. Verified live: run 2 leased three seconds after the fixed coordinator started.
• F13, observability, 0d67af9. federatedAvailability restated a quota refusal as worker health, so router health said stale heartbeat against a heartbeat one second old. Gates name themselves through router.ReasonedAvailability. Verified live.
• F14, authority, a5d361b. Gitea ingest never set acceptance. One recognized heading, bullets and checkboxes until the next heading, order preserved, section removed from the description. Verified live on run 3: five ordered items, prose above the heading kept as the description.
• F15, adapter, 1fd82f8. Two bugs. TmuxBackend.Prompt wrote the whole instruction with send-keys -l, Claude Code coalesced it into a paste, and the Enter was absorbed. And TaskLaunchAcknowledged meant "Prompt returned nil", not "the harness accepted it". Launch transport became a backend property, and ConfirmLaunch began polling for proof. The detection works. The fix did not: see F17.
• F16, lifecycle, 1888d42. renewLeases renewed whenever a session existed and PaneCapture succeeded, so a pane that opened and never started renewed forever. This is why the July task stayed orphaned: F15 explains why nothing started, F16 why the lease never let go. Renewal now needs the agent busy, or the pane capture to differ from the hash recorded at the previous renewal. lease.ProgressSHA carries that hash.
• F17, adapter, f54fb00. pendingInput scanned every line beginning with the prompt marker, but queued and already-accepted input renders with the same prefix. Only the editor owning the pane cursor is unsubmitted, so inputState reads #{cursor_y} and captures screen rows without -J, which would invalidate the row index. On that footing ConfirmLaunch became an active submit protocol: resend Enter while the live editor still holds exactly what was submitted, at most three times, no closer than two poll intervals, then observe until the deadline. Queued input confirms rather than fails. Evidence records confirmation, submit_attempts and both timestamps.
• F19, lifecycle, edff021. A task that reached the router's MaxAttempts was permanently terminal. TaskReleased only increments Attempt, TaskCorrected could not touch it, and no HTTP route emitted a correction. POST /v1/tasks/{id}/retry requires the task to be failed, unleased, and failed with reason: retry_limit, then appends one TaskCorrected naming that failure with state: queued and attempt: 0. Identity, goal, acceptance, decisions, phase and artifact refs all survive, and the original failures stay in the log. operation_id is required and makes it idempotent. Deliberately not a generic correction endpoint.

The finding that mattered most

Run 3 failed three times with prompt_not_submitted, and six isolated probes of the same code path could not reproduce it: fresh session, untrusted directory with the trust dialog, launch.md present before startup, 100ms readiness polling, env -i with only the worker's systemd variables, and the real StartAgent/Prompt/ConfirmLaunch against a byte-identical git worktree. All six submitted on the first Enter.

F17 turned that disagreement into data instead of a theory. The first live launch after it deployed:

launch 06G44JZB80MZBEY97196EZN8EC confirmed: confirmation=editor_cleared
submit_attempts=2 first_submit_at=2026-08-27T09:12:00.801855937Z
confirmed_at=2026-08-27T09:12:01.315587139Z

Production needs the second Enter. Isolated probes need one. The submit is not deterministic, which is exactly what F15's ConfirmLaunch comment asserted it was.

Still open

• F18, observability. A queued launch used to be misread as unsubmitted. F17 fixes the predicate, but nothing else that parses pane text distinguishes the active editor from history, so the same class of error can recur wherever PaneCapture output is matched.
• F9 is unchanged by F19. An operator still cannot release a lease someone else owns without supplying that owner's harness_id and lease_epoch, which means reading them out of the event log first. F19 recovers a terminal task, which is a different escape hatch.
• F6, observability. Worker health reported active_task: null while the store showed the task leased. Run 2 showed this was partly honest, because the worker really had no working agent. Recheck now that launches confirm.
• F4, adapter quality. harness "opencode" has no session-file resolver, so activity and thrash triggers are degraded on opencode.
• F10, hygiene. A coordinator-side release under a live worker leaves the worker renewing a lease it no longer holds.
• Unset gated tokens: ORCHESTRA_MCP_TOKEN, ORCHESTRA_MAVEN_TOKEN, ORCHESTRA_AGENT_TOKEN.
• gofmt -l fails on internal/provider/provider.go, internal/router/router.go and internal/webui/webui.go at edff021. None were touched by these runs. internal/router/router.go picked it up in 0d67af9. go vet passes, which is why nobody noticed.

Operator actions taken by hand, and why

• Task 06G3ZCZWJ3QHF992ZMDGSJ0PYG, run 2, was blocked with block_reason: operator_block rather than released. A released task returns to queued, and the router leases queued tasks, so releasing it would have put it in competition with run 3.
• The first block attempt returned task version conflict. Store.validateTransition fences every lifecycle event on a leased task, so the payload needs harness_id and lease_epoch even though the HTTP handler does not ask for them. That is F9 in practice.
• Task 06G44JZB80MZBEY97196EZN8EC, run 3, was recovered with F19's retry rather than by filing a fourth issue. That keeps kami/test-e2e#3 served by its own task and proves terminal recovery in the same run.

Deployment boundary

Both halves report edff021, built from a detached worktree of that revision. Worker sha256 68ac265455acb0e007bfb0e898a91b317cd7289d7f2bf5da6df1224e1d6706a5 at /usr/local/bin/orchestra-worker. Coordinator image built 13:08:30 +0400.