orchestra/BURNIN.md

Burn-in: live conformance, not more architecture

Written 2026-08-26. The workflow is feature-complete enough to exercise. What remains is empirical: run the same real task shape through each harness and classify what breaks. Isolated tests are insufficient by design here, and only the live owner path establishes conformance.

Do not add workflow features while this is running. Findings decide the next implementation work.

Deployment state, probed 2026-08-26

Burn-in cannot start yet. Two blockers, one config gap.

Fact	Evidence
The API is up on homesrv	GET 192.168.1.104:9145/readyz → ready:true, store/router/gitea/jsonl all ready
The workpc worker is up and serving two harnesses	orchestra-worker pid 741, restarted 2026-08-26 11:39, no errors since
The deployed worker predates every v3 unit	go version -m /usr/local/bin/orchestra-worker → vcs.revision=97a9c65, vcs.time=2026-07-30
The v3 work is uncommitted	Both sessions' units are working-tree changes on webui-and-audit-reconciliation
Codex has no harness entry at all	/etc/orchestra/harnesses.json declares only workpc-claude (tmux) and workpc-opencode (herdr)
No homesrv worker is running	Only the API container runs there; the three homesrv-* herdrs have no worker

Consequences:

1. Commit, then rebuild and redeploy both artifacts. The API container (docker compose -f compose.yaml -f compose.live.yaml up -d --build in /mnt/server/home/kami/docker-apps/orchestra-web-ui) and the worker binary (staged at workpc:~/orchestra-deploy/orchestra-worker, then installed to /usr/local/bin, then systemctl restart orchestra-worker). Confirm the API revision at GET /v1/admin/diagnostics and the worker's at go version -m. Neither currently has phases, review, submission, the agent surface, or the reconcile escalation.
2. Codex cannot be burned in until it has a harnesses.json entry. Three of four target harnesses are unreachable today: codex is unconfigured, and both homesrv-claude/homesrv-codex/homesrv-opencode have no worker.
3. The registry is misleading about herdr transports. No entry in the live config.jsonc sets backend or address, so all six resolve to <machine>:9245 (registry.defaultHerdrPort), and both ports are closed. The workpc worker actually uses a tmux socket and a unix socket (/home/kami/.config/herdr/herdr.sock). Federated reachability defers to worker heartbeat, so this does not block leasing, but a TCP probe of 9245 is not evidence about any of these harnesses. Do not diagnose from it.

Evidence to record per run

One row per run. The launch instruction is now dumped to <worktree>/.orchestra/launch.md (herdr.LaunchContextFile) at every launch, local and federated, so the context is auditable without reading pane scrollback.

harness            (claude | codex | opencode)
task id
lease epoch        (every epoch, if the task rotates)
session ids        (pane ids per epoch)
human decision ids
launch context     (sha256 of .orchestra/launch.md, per launch)
handoff ref
git sha            (at every boundary: launch, each handoff, review, submit, merge)
review ref
submission ref
pr id
completion receipt

The five flows

1. Boring success. task → research → plan → implement → review → pr → merge.
2. Mid-session correction. Agent is doing A, the human says B, the next verified boundary delivers B, and A becomes history rather than a competing instruction.
3. Rotation. Agent A hands off, agent B resumes. Switch harnesses across the boundary if both are up.
4. Human rejection. Submit sha A, comment on the pull request, task reopens, fix at sha B, a fresh review, the same pull request, merge.
5. Failure path. Human source goes down, the reconcile streak escalates, the session hands off, the successor lease is refused, the source recovers, and a corrected successor starts.

What to inspect by hand

For every launch, read .orchestra/launch.md and ask only:

does this agent know
- what the task actually wants?
- what was most recently decided?
- what phase it is in?
- what earlier material is merely historical?
- what to do next?

Then compare that against what the model did.

Classify before fixing

Every failure gets exactly one label before any code is written:

authority bug            the wrong thing outranked the right thing
context-selection bug    the renderer showed or hid the wrong material
lifecycle bug            state, lease, phase or event handling is wrong
adapter/harness bug       the pane, occupancy, boundary or prompt path is wrong
model-following failure  the context was right and the model ignored it
operator-policy gap      credentials, deployment or configuration

This matters because the cheap response to every failure is another prompt rule, and prompt rules accumulated to compensate for lifecycle or adapter bugs are how the enforcement boundary rots. A model-following failure is the only class a prompt change should ever answer.

Pane credential cleanup (operator, in parallel)

The agent pane inherits the worker's environment. On workpc the worker is orchestra-worker.service, User=kami, EnvironmentFile=/etc/orchestra/worker.env (mode 0600, root-owned, deliberately unreadable here). Remove from that file, and from anything else the pane inherits:

gitea mutation token
vikunja mutation credentials
orchestra operator / system token

Retain only ORCHESTRA_AGENT_TOKEN and credentials a specific task genuinely needs. If an agent needs a forge operation, it asks Orchestra to perform it.

Then verify from inside a real pane:

git push                 # must fail unless Orchestra supplied the auth
curl .../v1/tasks/<id>/submission   # must be 403 for the agent surface
tea pr create            # must have no usable mutation credential

Note what is not enforced by code: nothing in this repo scrubs the pane environment, and a fake git earlier in $PATH is not enforcement because /usr/bin/git bypasses it. Credential isolation is the enforcement. Confinement of network, filesystem and destructive commands belongs to whatever launches the process (herdr, a container, systemd, bubblewrap), not to Orchestra, which supplies identity and policy inputs.

Exit criterion

Roughly 10 to 20 real tasks, with the five flows covered on each harness that is actually up. Then read the classified failures and decide the next implementation unit from them.