diff --git a/HANDOFF-2026-08-27-burnin.md b/HANDOFF-2026-08-27-burnin.md new file mode 100644 index 0000000..5ac3014 --- /dev/null +++ b/HANDOFF-2026-08-27-burnin.md @@ -0,0 +1,159 @@ +# Handoff — burn-in, run 2 diagnosed, run 3 blocked on an operator step + +Written 2026-08-27, early. Read with `HANDOFF-2026-08-26-burnin.md` (the +previous session), `BURNIN.md`, `AUDIT.md` and `CLAUDE.md`. Everything below +was observed live. Nothing here is inferred from a passing test. + +Run 2 was not a conformance run. It was worth more than one: it found three +blocking bugs in a row, each hidden behind the previous one. + +## Deployed state, and the mismatch + +| Half | Live revision | Built and staged | Evidence | +|---|---|---|---| +| Coordinator, homesrv container | `0d67af9` | image for `1fd82f8` built, not recreated | `docker logs orchestra-api` startup line | +| Worker, workpc systemd | `77a2b32` | `~/orchestra-deploy/orchestra-worker.1fd82f8` | `journalctl -u orchestra-worker` | + +Branch `webui-and-audit-reconciliation`, HEAD `1fd82f8`. + +Staged worker sha256 `19e17893d6f87651050c9826e8264aba313d2ed4385013efd4576e13d5b7c9ab`. +The installed `/usr/local/bin/orchestra-worker` is `2a850f2d1102d390fe5960011756e14cd14ed14f6e15384189e63e961eb419b1`. + +Neither half is at `1fd82f8`. Run 3 cannot start until both are. + +## Do this first + +Installing the worker needs root, so the sequence starts with the operator. +It must run in this order, and the reason is F16 below. + +1. Operator stops `orchestra-worker`. This is what stops the stale renewal. +2. Kill only the run 2 tmux session: + `tmux -L orchestra kill-session -t orchestra-06g3zczwj3qhf992zmdgsj0pyg-6b87ef68`. + Nothing else on that socket. Other sessions there are not Orchestra's. +3. Operator installs `~/orchestra-deploy/orchestra-worker.1fd82f8` to + `/usr/local/bin/orchestra-worker` and starts the unit. Verify the startup + line reads revision `1fd82f8`. +4. Recreate the coordinator. The image is already built: + `cd /home/kami/docker-apps/orchestra-web-ui && docker compose -f compose.yaml -f compose.live.yaml up -d --no-deps --force-recreate orchestra-api` +5. Verify both revisions and a fresh heartbeat at + `GET /v1/federation/workers` before creating anything. +6. Decide what happens to task `06G3ZCZWJ3QHF992ZMDGSJ0PYG` (see below), then + create the run 3 issue. + +## The open decision + +Task `06G3ZCZWJ3QHF992ZMDGSJ0PYG` (issue `kami/test-e2e#2`) is still leased to +epoch `06G3ZGYSFGWKSV9CHJ4EADQVBC` and is at version 5. Once the worker stops +renewing it, the lease expires and the router will re-lease it under the fixed +worker. That would be a genuine F15 test. It also competes with run 3 for a +single-capacity worker, and it is not a fresh issue. + +The operator was asked and had not answered when this session ended. Ask again +before creating run 3. + +## What run 2 proved + +Autonomous assignment works. The lease landed three seconds after the fixed +coordinator started, with no manual intervention: + +``` +306 TaskLeased 20:29:54 workpc-claude epoch 06G3ZGYSFGWKSV9CHJ4EADQVBC +307 TaskLaunchAcknowledged 20:30:00 pane orchestra-06g3zczwj3qhf992zmdgsj0pyg-6b87ef68:1.0 +308 TaskLeaseRenewed 20:49:59 same epoch +309 TaskLeaseRenewed 21:10:04 same epoch +``` + +`launch.md` carried a body-derived goal, the authority order, the ambiguity +ladder, the `frame` phase brief and a verified git state. + +And then nothing ran. Events 308 and 309 are a lease renewing itself around an +agent that never started. + +## Fixed this session + +- **F12**, lifecycle, was blocking every autonomous lease. `Store.QuotaSince` + reported an empty window as unknown, `QuotaAvailability` fails closed on + unknown, and every herdr in `config.jsonc` declares a quota limit. The only + producer of a receipt is a completed lease, so nothing could ever be leased. + The live event log had zero `QuotaReported` events. An empty window is now + observable zero. A receipt that declares its own consumption unknown still + fails closed. `0d67af9`. +- **F13**, observability. `federatedAvailability` restated a quota refusal as + worker health, so router health said `stale heartbeat` against a heartbeat + one second old. Gates now name themselves through `router.ReasonedAvailability`. + `0d67af9`. +- **F14**, authority. Gitea ingest never set `acceptance`. One recognized + heading (`acceptance` or `acceptance criteria`), bullets and checkboxes until + the next heading, order preserved, section removed from the description. No + section means no acceptance, which is correct rather than a failure. + `a5d361b`. +- **F15**, adapter and lifecycle, found live. Two bugs. `TmuxBackend.Prompt` + wrote the whole instruction with `send-keys -l`, Claude Code coalesced it + into a paste, and the following Enter was absorbed. And + `TaskLaunchAcknowledged` meant "Prompt returned nil", not "the harness + accepted it". Launch transport is now a backend property: claude on tmux + submits one line pointing at `.orchestra/launch.md`, everything else stays + inline. `ConfirmLaunch` polls until the input editor clears and the agent is + observably busy or blocked. An editor still holding the prompt at the + deadline returns `herdr.ErrPromptNotSubmitted`, the worker kills the pane and + drops the session, and `classifyLaunchError` returns `prompt_not_submitted` + instead of `launch_uncertain`. `1fd82f8`. + +`go build ./...`, `go vet ./...` and `go test ./...` all pass at `1fd82f8`. +F12 and F13 were verified live. F14 and F15 have tests only. + +## Open findings, in the order this session would take them + +- **F16**, lifecycle, new and unfixed. `renewLeases` + (`cmd/orchestra-worker/main.go:951`) renews whenever a session exists and + `PaneCapture` succeeds. Nothing requires the agent to have accepted a prompt + or done any work, so a pane that opened and never started renews its lease + forever. This is the mechanism behind the July stuck task: F15 explains why + nothing started, F16 explains why the lease never let go. A fix should + require observable progress, not pane existence. F15's `ConfirmLaunch` gives + the shape to reuse. +- **F6**, observability. Worker health reports `active_task: null` while the + store shows the task leased. Run 2 showed this was partly honest: the worker + really had no working agent. Recheck after F15 is live before treating it as + a reporting bug. +- **F4**, adapter quality. `harness "opencode" has no session-file resolver`, + so activity and thrash triggers are degraded on opencode. +- **F9**, lifecycle. Releasing a leased task through + `POST /v1/tasks//release` requires `harness_id` and `lease_epoch`. There + is no operator escape hatch. Run 2 made this concrete: with F16 renewing + forever, there was no way to end the run without stopping the worker. +- **F10**, hygiene. A coordinator-side release under a live worker leaves the + worker renewing a lease it no longer holds. +- **Unset gated tokens**: `ORCHESTRA_MCP_TOKEN`, `ORCHESTRA_MAVEN_TOKEN` and + `ORCHESTRA_AGENT_TOKEN` are still empty. + +## Things that will bite + +- **Another session still owns 18 uncommitted paths.** `internal/authn/*`, + `web/*`, `internal/webui/assets/*`, `AGENTS.md`, `deploy/DEPLOYMENT.md`, + `deploy/orchestra.env.example`, `AUDIT.md`, and the `deploy/build.sh` + extension that also builds `cmd/orchestra-user`. Commit by path. Never + `git add -A`. This session did not touch them, including `AUDIT.md`, which + is why the findings above are not in the ledger yet. +- **`deploy/build.sh` is one of those uncommitted files.** It works and this + session used it, but it is not in any commit. +- **Do not use `docker compose build`.** Its context is the shared live + checkout. Build from a detached worktree of the revision, as in + `HANDOFF-2026-08-26-burnin.md`. +- **This machine is workpc.** `hostname` is `bugmachine`. The worker, the + tmux socket `orchestra`, `/tmp/test-e2e` and `/tmp/test-e2e-worktrees` are + all local here. There is no SSH key from here to `192.168.1.105`, and none + is needed. homesrv is reachable at `kami@192.168.1.104` with `/usr/bin/ssh`; + the `ssh` on PATH is kitty's interactive-only wrapper and will fail. +- **`/tmp/test-e2e` survived**, head `faba139`, non-bare clone. `/tmp` is + tmpfs with a 10-day sweep, so check it exists before every run. +- **The real script is `scripts/orchestra_e2e_healthcheck.sh`**, 17 lines, + one `--help` flag, no per-check structure. Write issue bodies against what + is actually in the repo. +- **The MCP surface is a read-only diagnostic with no credential.** + `curl -H 'X-Orchestra-Surface: mcp' http://192.168.1.104:9145/v1/tasks` is + how this session read live state. Mutations need the TUI token from `.env`. +- **`secret-guard` blocks reading `.env`.** Ask the operator for the one value + needed, or have them run the command. +- **The three queued `correx` tasks will never lease.** No worker declares + that project. Router health says so.