diff --git a/BURNIN.md b/BURNIN.md index 61752d6..6c48ec0 100644 --- a/BURNIN.md +++ b/BURNIN.md @@ -486,3 +486,127 @@ F5 exposed done F8 fixed done fresh issue, no manual leasing ``` + +## Runs 2 and 3, 2026-08-27 + +Written from live evidence. Neither run was a conformance pass, and both were +worth more than one: run 2 found three blocking bugs behind each other, and +run 3 found two more plus the reason six isolated probes disagreed with +production. + +`AUDIT.md` is uncommitted and owned by another session, so the burn-in ledger +lives here. + +### Fixed, with the revision each landed in + +- **F12**, lifecycle, `0d67af9`. `Store.QuotaSince` reported an empty window as + unknown, `QuotaAvailability` fails closed on unknown, and every herdr in + `config.jsonc` declares a quota limit. The only producer of a receipt is a + completed lease, so nothing could ever be leased. An empty window is now + observable zero. A receipt that declares its own consumption unknown still + fails closed. Verified live: run 2 leased three seconds after the fixed + coordinator started. +- **F13**, observability, `0d67af9`. `federatedAvailability` restated a quota + refusal as worker health, so router health said `stale heartbeat` against a + heartbeat one second old. Gates name themselves through + `router.ReasonedAvailability`. Verified live. +- **F14**, authority, `a5d361b`. Gitea ingest never set `acceptance`. One + recognized heading, bullets and checkboxes until the next heading, order + preserved, section removed from the description. Verified live on run 3: five + ordered items, prose above the heading kept as the description. +- **F15**, adapter, `1fd82f8`. Two bugs. `TmuxBackend.Prompt` wrote the whole + instruction with `send-keys -l`, Claude Code coalesced it into a paste, and + the Enter was absorbed. And `TaskLaunchAcknowledged` meant "Prompt returned + nil", not "the harness accepted it". Launch transport became a backend + property, and `ConfirmLaunch` began polling for proof. The detection works. + The fix did not: see F17. +- **F16**, lifecycle, `1888d42`. `renewLeases` renewed whenever a session + existed and `PaneCapture` succeeded, so a pane that opened and never started + renewed forever. This is why the July task stayed orphaned: F15 explains why + nothing started, F16 why the lease never let go. Renewal now needs the agent + busy, or the pane capture to differ from the hash recorded at the previous + renewal. `lease.ProgressSHA` carries that hash. +- **F17**, adapter, `f54fb00`. `pendingInput` scanned every line beginning with + the prompt marker, but queued and already-accepted input renders with the + same prefix. Only the editor owning the pane cursor is unsubmitted, so + `inputState` reads `#{cursor_y}` and captures screen rows without `-J`, which + would invalidate the row index. On that footing `ConfirmLaunch` became an + active submit protocol: resend Enter while the live editor still holds + exactly what was submitted, at most three times, no closer than two poll + intervals, then observe until the deadline. Queued input confirms rather than + fails. Evidence records `confirmation`, `submit_attempts` and both timestamps. +- **F19**, lifecycle, `edff021`. A task that reached the router's + `MaxAttempts` was permanently terminal. `TaskReleased` only increments + `Attempt`, `TaskCorrected` could not touch it, and no HTTP route emitted a + correction. `POST /v1/tasks/{id}/retry` requires the task to be failed, + unleased, and failed with `reason: retry_limit`, then appends one + `TaskCorrected` naming that failure with `state: queued` and `attempt: 0`. + Identity, goal, acceptance, decisions, phase and artifact refs all survive, + and the original failures stay in the log. `operation_id` is required and + makes it idempotent. Deliberately not a generic correction endpoint. + +### The finding that mattered most + +Run 3 failed three times with `prompt_not_submitted`, and six isolated probes +of the same code path could not reproduce it: fresh session, untrusted +directory with the trust dialog, `launch.md` present before startup, 100ms +readiness polling, `env -i` with only the worker's systemd variables, and the +real `StartAgent`/`Prompt`/`ConfirmLaunch` against a byte-identical git +worktree. All six submitted on the first Enter. + +F17 turned that disagreement into data instead of a theory. The first live +launch after it deployed: + +``` +launch 06G44JZB80MZBEY97196EZN8EC confirmed: confirmation=editor_cleared +submit_attempts=2 first_submit_at=2026-08-27T09:12:00.801855937Z +confirmed_at=2026-08-27T09:12:01.315587139Z +``` + +Production needs the second Enter. Isolated probes need one. The submit is not +deterministic, which is exactly what F15's `ConfirmLaunch` comment asserted it +was. + +### Still open + +- **F18**, observability. A queued launch used to be misread as unsubmitted. + F17 fixes the predicate, but nothing else that parses pane text distinguishes + the active editor from history, so the same class of error can recur wherever + `PaneCapture` output is matched. +- **F9** is unchanged by F19. An operator still cannot release a lease someone + else owns without supplying that owner's `harness_id` and `lease_epoch`, + which means reading them out of the event log first. F19 recovers a terminal + task, which is a different escape hatch. +- **F6**, observability. Worker health reported `active_task: null` while the + store showed the task leased. Run 2 showed this was partly honest, because + the worker really had no working agent. Recheck now that launches confirm. +- **F4**, adapter quality. `harness "opencode" has no session-file resolver`, + so activity and thrash triggers are degraded on opencode. +- **F10**, hygiene. A coordinator-side release under a live worker leaves the + worker renewing a lease it no longer holds. +- Unset gated tokens: `ORCHESTRA_MCP_TOKEN`, `ORCHESTRA_MAVEN_TOKEN`, + `ORCHESTRA_AGENT_TOKEN`. +- `gofmt -l` fails on `internal/provider/provider.go`, + `internal/router/router.go` and `internal/webui/webui.go` at `edff021`. None + were touched by these runs. `internal/router/router.go` picked it up in + `0d67af9`. `go vet` passes, which is why nobody noticed. + +### Operator actions taken by hand, and why + +- Task `06G3ZCZWJ3QHF992ZMDGSJ0PYG`, run 2, was blocked with + `block_reason: operator_block` rather than released. A released task returns + to `queued`, and the router leases `queued` tasks, so releasing it would have + put it in competition with run 3. +- The first block attempt returned `task version conflict`. + `Store.validateTransition` fences every lifecycle event on a leased task, so + the payload needs `harness_id` and `lease_epoch` even though the HTTP handler + does not ask for them. That is F9 in practice. +- Task `06G44JZB80MZBEY97196EZN8EC`, run 3, was recovered with F19's retry + rather than by filing a fourth issue. That keeps `kami/test-e2e#3` served by + its own task and proves terminal recovery in the same run. + +### Deployment boundary + +Both halves report `edff021`, built from a detached worktree of that revision. +Worker sha256 `68ac265455acb0e007bfb0e898a91b317cd7289d7f2bf5da6df1224e1d6706a5` +at `/usr/local/bin/orchestra-worker`. Coordinator image built 13:08:30 +0400.