From 801ec1be242fd0196a8021d24e16702a774712c0 Mon Sep 17 00:00:00 2001 From: kami Date: Thu, 27 Aug 2026 13:21:31 +0400 Subject: [PATCH] Record runs 2 and 3 in the burn-in ledger F12 through F19, each against the revision it landed in, with what was verified live and what has tests only. AUDIT.md is uncommitted and owned by another session, so the ledger goes here rather than colliding with it. The entry worth keeping: run 3 failed three times with prompt_not_submitted while six isolated probes of the same code path submitted on the first Enter. F17's evidence line settles it, submit_attempts=2 in production against 1 in every probe. Co-Authored-By: Claude Opus 5 --- BURNIN.md | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) diff --git a/BURNIN.md b/BURNIN.md index 61752d6..6c48ec0 100644 --- a/BURNIN.md +++ b/BURNIN.md @@ -486,3 +486,127 @@ F5 exposed done F8 fixed done fresh issue, no manual leasing ``` + +## Runs 2 and 3, 2026-08-27 + +Written from live evidence. Neither run was a conformance pass, and both were +worth more than one: run 2 found three blocking bugs behind each other, and +run 3 found two more plus the reason six isolated probes disagreed with +production. + +`AUDIT.md` is uncommitted and owned by another session, so the burn-in ledger +lives here. + +### Fixed, with the revision each landed in + +- **F12**, lifecycle, `0d67af9`. `Store.QuotaSince` reported an empty window as + unknown, `QuotaAvailability` fails closed on unknown, and every herdr in + `config.jsonc` declares a quota limit. The only producer of a receipt is a + completed lease, so nothing could ever be leased. An empty window is now + observable zero. A receipt that declares its own consumption unknown still + fails closed. Verified live: run 2 leased three seconds after the fixed + coordinator started. +- **F13**, observability, `0d67af9`. `federatedAvailability` restated a quota + refusal as worker health, so router health said `stale heartbeat` against a + heartbeat one second old. Gates name themselves through + `router.ReasonedAvailability`. Verified live. +- **F14**, authority, `a5d361b`. Gitea ingest never set `acceptance`. One + recognized heading, bullets and checkboxes until the next heading, order + preserved, section removed from the description. Verified live on run 3: five + ordered items, prose above the heading kept as the description. +- **F15**, adapter, `1fd82f8`. Two bugs. `TmuxBackend.Prompt` wrote the whole + instruction with `send-keys -l`, Claude Code coalesced it into a paste, and + the Enter was absorbed. And `TaskLaunchAcknowledged` meant "Prompt returned + nil", not "the harness accepted it". Launch transport became a backend + property, and `ConfirmLaunch` began polling for proof. The detection works. + The fix did not: see F17. +- **F16**, lifecycle, `1888d42`. `renewLeases` renewed whenever a session + existed and `PaneCapture` succeeded, so a pane that opened and never started + renewed forever. This is why the July task stayed orphaned: F15 explains why + nothing started, F16 why the lease never let go. Renewal now needs the agent + busy, or the pane capture to differ from the hash recorded at the previous + renewal. `lease.ProgressSHA` carries that hash. +- **F17**, adapter, `f54fb00`. `pendingInput` scanned every line beginning with + the prompt marker, but queued and already-accepted input renders with the + same prefix. Only the editor owning the pane cursor is unsubmitted, so + `inputState` reads `#{cursor_y}` and captures screen rows without `-J`, which + would invalidate the row index. On that footing `ConfirmLaunch` became an + active submit protocol: resend Enter while the live editor still holds + exactly what was submitted, at most three times, no closer than two poll + intervals, then observe until the deadline. Queued input confirms rather than + fails. Evidence records `confirmation`, `submit_attempts` and both timestamps. +- **F19**, lifecycle, `edff021`. A task that reached the router's + `MaxAttempts` was permanently terminal. `TaskReleased` only increments + `Attempt`, `TaskCorrected` could not touch it, and no HTTP route emitted a + correction. `POST /v1/tasks/{id}/retry` requires the task to be failed, + unleased, and failed with `reason: retry_limit`, then appends one + `TaskCorrected` naming that failure with `state: queued` and `attempt: 0`. + Identity, goal, acceptance, decisions, phase and artifact refs all survive, + and the original failures stay in the log. `operation_id` is required and + makes it idempotent. Deliberately not a generic correction endpoint. + +### The finding that mattered most + +Run 3 failed three times with `prompt_not_submitted`, and six isolated probes +of the same code path could not reproduce it: fresh session, untrusted +directory with the trust dialog, `launch.md` present before startup, 100ms +readiness polling, `env -i` with only the worker's systemd variables, and the +real `StartAgent`/`Prompt`/`ConfirmLaunch` against a byte-identical git +worktree. All six submitted on the first Enter. + +F17 turned that disagreement into data instead of a theory. The first live +launch after it deployed: + +``` +launch 06G44JZB80MZBEY97196EZN8EC confirmed: confirmation=editor_cleared +submit_attempts=2 first_submit_at=2026-08-27T09:12:00.801855937Z +confirmed_at=2026-08-27T09:12:01.315587139Z +``` + +Production needs the second Enter. Isolated probes need one. The submit is not +deterministic, which is exactly what F15's `ConfirmLaunch` comment asserted it +was. + +### Still open + +- **F18**, observability. A queued launch used to be misread as unsubmitted. + F17 fixes the predicate, but nothing else that parses pane text distinguishes + the active editor from history, so the same class of error can recur wherever + `PaneCapture` output is matched. +- **F9** is unchanged by F19. An operator still cannot release a lease someone + else owns without supplying that owner's `harness_id` and `lease_epoch`, + which means reading them out of the event log first. F19 recovers a terminal + task, which is a different escape hatch. +- **F6**, observability. Worker health reported `active_task: null` while the + store showed the task leased. Run 2 showed this was partly honest, because + the worker really had no working agent. Recheck now that launches confirm. +- **F4**, adapter quality. `harness "opencode" has no session-file resolver`, + so activity and thrash triggers are degraded on opencode. +- **F10**, hygiene. A coordinator-side release under a live worker leaves the + worker renewing a lease it no longer holds. +- Unset gated tokens: `ORCHESTRA_MCP_TOKEN`, `ORCHESTRA_MAVEN_TOKEN`, + `ORCHESTRA_AGENT_TOKEN`. +- `gofmt -l` fails on `internal/provider/provider.go`, + `internal/router/router.go` and `internal/webui/webui.go` at `edff021`. None + were touched by these runs. `internal/router/router.go` picked it up in + `0d67af9`. `go vet` passes, which is why nobody noticed. + +### Operator actions taken by hand, and why + +- Task `06G3ZCZWJ3QHF992ZMDGSJ0PYG`, run 2, was blocked with + `block_reason: operator_block` rather than released. A released task returns + to `queued`, and the router leases `queued` tasks, so releasing it would have + put it in competition with run 3. +- The first block attempt returned `task version conflict`. + `Store.validateTransition` fences every lifecycle event on a leased task, so + the payload needs `harness_id` and `lease_epoch` even though the HTTP handler + does not ask for them. That is F9 in practice. +- Task `06G44JZB80MZBEY97196EZN8EC`, run 3, was recovered with F19's retry + rather than by filing a fourth issue. That keeps `kami/test-e2e#3` served by + its own task and proves terminal recovery in the same run. + +### Deployment boundary + +Both halves report `edff021`, built from a detached worktree of that revision. +Worker sha256 `68ac265455acb0e007bfb0e898a91b317cd7289d7f2bf5da6df1224e1d6706a5` +at `/usr/local/bin/orchestra-worker`. Coordinator image built 13:08:30 +0400.