diff --git a/HANDOFF-2026-08-27-burnin-2.md b/HANDOFF-2026-08-27-burnin-2.md new file mode 100644 index 0000000..f9c57ed --- /dev/null +++ b/HANDOFF-2026-08-27-burnin-2.md @@ -0,0 +1,167 @@ +# Handoff — run 3 relaunch pending, two hard predicates open + +Written 2026-08-27, 11:00 UTC, mid-observation. Read with +`HANDOFF-2026-08-27-burnin.md` (the session before this one), `BURNIN.md` +(the ledger, now current through run 3), `AUDIT.md` and `CLAUDE.md`. + +Everything below was observed live unless it says otherwise. + +## Evidence standard, set by the operator + +Use this hierarchy and do not mix the tiers: + +``` +event log → lifecycle truth +worker journal → worker-local observations and confirmer receipts +pane capture → harness evidence only +code-path argument → supporting evidence, not live proof +``` + +This exists because this session reported "F16 passes its first branch live" +on the strength of three `TaskLeaseRenewed` events. The predicate was inert. +A code-path argument was presented as live proof, and the bug survived the +report. Do not repeat that. + +## Deployed state + +| Half | Revision | Evidence | +|---|---|---| +| Coordinator, homesrv container | `edbe98f` | `docker logs orchestra-api`, 10:45:46 UTC | +| Worker, workpc systemd | `edbe98f` | `journalctl -u orchestra-worker`, 14:45:26 +04 | + +Branch `webui-and-audit-reconciliation`, HEAD `edbe98f`. Worker sha256 +`e94ab4ddbbd755796ca68e273a55029895ec930ea88405be2f6af65be9bdc494`, staged at +`~/orchestra-deploy/orchestra-worker.edbe98f`. + +Both halves match. Do not start a new run until they do again after any change. + +## What is being watched right now + +Task `06G44JZB80MZBEY97196EZN8EC`, issue `kami/test-e2e#3`, is `leased` at v15 +under **stranded epoch `06G44ZBZX4YH6PRN7Y4ZH8GG3W`**. Its pane is gone and its +agent has been idle since 09:13. The last renewal was seq 358 at 10:32:09, by +the old worker. The lease runs to roughly 11:02 UTC. + +Nobody should touch it. The expiry is the test. + +Expected chain, autonomous, no operator action: + +``` +renewal boundary reached +→ progress observation fails, pane is gone +→ worker does not renew +→ lease expires +→ TaskReleased(reason=lease_expired) +→ queued, attempt 1 +→ router selects the worker +→ new lease, new epoch, new pane +→ confirmed input submission +→ TaskLaunchAcknowledged +``` + +### Two hard failure predicates + +``` +old epoch 06G44ZBZX4YH6PRN7Y4ZH8GG3W renewed after 10:45:26 → F16 fail +first orchestra-originated post-frame input has no confirmer receipt → F20 fail +``` + +The F20 receipt is strictly this line in the worker journal: + +``` +input to confirmed: confirmation= submit_attempts= ... +``` + +Pane movement, text leaving the editor, and later model output are all +insufficient. + +### Checks to capture, in order + +1. No `TaskLeaseRenewed` carrying the stranded epoch after 10:45:26. +2. Release reason is literally `lease_expired`, not a restart or reconcile + fallback. +3. New lease epoch differs from `06G44ZBZX4YH6PRN7Y4ZH8GG3W`. +4. `ProgressSHA`, `UsageBaseline` and `PickupAcknowledged` start fresh for the + new epoch. State this precisely: the reset is **structurally established by + the reconciler**, and the first successful renewal under the new epoch shows + the reconstructed lease stayed usable. Those fields are not exposed, so do + not claim their runtime values were observed. +5. Launch confirmation evidence carries `confirmation=` and `submit_attempts=`. +6. No operator action between expiry and relaunch. +7. Then the F20 boundary: the first Orchestra-originated input after `frame` + completes. + +If the task reaches real implementation after that with no manual +intervention, this stops being a diagnostic run. + +## Fixed since the last handoff + +Each landed in the revision named. `BURNIN.md` carries the full entries. + +- **F16**, lifecycle, `1888d42` then `770cc6a` then `edbe98f`. Renewal requires + observable progress. It shipped **inert**: `reconcileLeases` rebuilt every + lease from the coordinator's task list each tick and dropped `ProgressSHA`, + so the check took its no-baseline branch forever. `UsageBaseline` and + `PickupAcknowledged` were wiped the same way. Progress is now measured with + input lines removed, so keystrokes are not work. +- **F17**, adapter, `f54fb00`. `pendingInput` scanned every line with the + prompt marker, but queued and accepted input renders identically. Only the + editor owning the cursor is unsubmitted. `ConfirmLaunch` became an active + submit protocol: resend Enter while the live editor holds exactly what was + submitted, at most 3 times, then observe. Verified live: + `submit_attempts=2` in production against 1 in six isolated probes. +- **F19**, lifecycle, `edff021`. `POST /v1/tasks/{id}/retry`, full-control + surfaces only, for a failed and unleased task whose failure reason is + `retry_limit`. Requires `operation_id` and is idempotent. Verified live: the + task leased itself one second after the correction. +- **F20**, adapter, `edbe98f`. Every Orchestra-originated pane write confirms. + `LaunchConfirmer` is `InputConfirmer`, `ConfirmLaunch` is `ConfirmInput`, and + `sendPrompt` and `sendLine` both route through it. Tests only so far. +- **F12**, **F13**, **F14** landed earlier and are described in `BURNIN.md`. + F12 and F13 were verified live, F14 on run 3's ingest. + +## Open + +- **F20 live**, unproven. This is the next boundary. +- **F18**, observability. Nothing besides `inputState` distinguishes the active + editor from history when parsing pane text, so the same class of error can + recur wherever `PaneCapture` output is matched. +- **F9**, unchanged. An operator cannot release a lease someone else owns + without supplying that owner's `harness_id` and `lease_epoch`, read out of + the event log first. F19 did not address this. +- **Approval input is unconfirmed on purpose.** `main.go:1078` writes `y`, `n` + and `ENTER` to a y/n dialog, not an editor, so the editor predicate would + report nonsense. It needs its own predicate if it is ever covered. +- **F6**, observability. Worker health reported `active_task: null` against a + leased task. Recheck now that launches confirm. +- **F4**, adapter quality. `harness "opencode" has no session-file resolver`. +- **F10**, hygiene. A coordinator-side release under a live worker leaves the + worker renewing a lease it no longer holds. +- Unset gated tokens: `ORCHESTRA_MCP_TOKEN`, `ORCHESTRA_MAVEN_TOKEN`, + `ORCHESTRA_AGENT_TOKEN`. +- `gofmt -l` fails on `internal/provider/provider.go`, + `internal/router/router.go` and `internal/webui/webui.go`. Untouched by these + runs, and `go vet` passes, which is why nobody noticed. + +## Things that will bite + +- **Another session still owns 18 uncommitted paths**, including `AUDIT.md`. + Commit by path. Never `git add -A`. The burn-in ledger lives in `BURNIN.md` + for exactly this reason. +- **An unexplained line was typed into the run-3 pane.** The operator confirms + it was not them, and the worker journal shows no send. Treat unexplained pane + input as possible, which is why progress must ignore input entirely. +- **Do not use `docker compose build`.** Its context is the shared live + checkout. Build from a detached worktree, as `deploy/build.sh` does and as + the coordinator image command in `HANDOFF-2026-08-26-burnin.md` shows. +- **Installing the worker needs root**, so it is always the operator's step. + Mutations through the API need the TUI token from the compose `.env`, which + `secret-guard` blocks from being read. Hand the operator the command. +- **This machine is workpc**, `hostname` is `bugmachine`. homesrv is + `kami@192.168.1.104` with `/usr/bin/ssh`, not the `ssh` on PATH. +- **`/tmp` is tmpfs with a 10-day sweep.** Check `/tmp/test-e2e` and + `/tmp/test-e2e-worktrees` exist before every run. +- **The MCP surface is read-only and needs no credential.** + `curl -H 'X-Orchestra-Surface: mcp' http://192.168.1.104:9145/v1/tasks`. +- **Three queued `correx` tasks will never lease.** No worker declares that + project. Router health says so.