Record runs 2 and 3 in the burn-in ledger

F12 through F19, each against the revision it landed in, with what was
verified live and what has tests only. AUDIT.md is uncommitted and owned by
another session, so the ledger goes here rather than colliding with it.

The entry worth keeping: run 3 failed three times with prompt_not_submitted
while six isolated probes of the same code path submitted on the first Enter.
F17's evidence line settles it, submit_attempts=2 in production against 1 in
every probe.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-27 13:21:31 +04:00
parent edff021265
commit 801ec1be24
+124
View File
@@ -486,3 +486,127 @@ F5 exposed done
F8 fixed done
fresh issue, no manual leasing
```
## Runs 2 and 3, 2026-08-27
Written from live evidence. Neither run was a conformance pass, and both were
worth more than one: run 2 found three blocking bugs behind each other, and
run 3 found two more plus the reason six isolated probes disagreed with
production.
`AUDIT.md` is uncommitted and owned by another session, so the burn-in ledger
lives here.
### Fixed, with the revision each landed in
- **F12**, lifecycle, `0d67af9`. `Store.QuotaSince` reported an empty window as
unknown, `QuotaAvailability` fails closed on unknown, and every herdr in
`config.jsonc` declares a quota limit. The only producer of a receipt is a
completed lease, so nothing could ever be leased. An empty window is now
observable zero. A receipt that declares its own consumption unknown still
fails closed. Verified live: run 2 leased three seconds after the fixed
coordinator started.
- **F13**, observability, `0d67af9`. `federatedAvailability` restated a quota
refusal as worker health, so router health said `stale heartbeat` against a
heartbeat one second old. Gates name themselves through
`router.ReasonedAvailability`. Verified live.
- **F14**, authority, `a5d361b`. Gitea ingest never set `acceptance`. One
recognized heading, bullets and checkboxes until the next heading, order
preserved, section removed from the description. Verified live on run 3: five
ordered items, prose above the heading kept as the description.
- **F15**, adapter, `1fd82f8`. Two bugs. `TmuxBackend.Prompt` wrote the whole
instruction with `send-keys -l`, Claude Code coalesced it into a paste, and
the Enter was absorbed. And `TaskLaunchAcknowledged` meant "Prompt returned
nil", not "the harness accepted it". Launch transport became a backend
property, and `ConfirmLaunch` began polling for proof. The detection works.
The fix did not: see F17.
- **F16**, lifecycle, `1888d42`. `renewLeases` renewed whenever a session
existed and `PaneCapture` succeeded, so a pane that opened and never started
renewed forever. This is why the July task stayed orphaned: F15 explains why
nothing started, F16 why the lease never let go. Renewal now needs the agent
busy, or the pane capture to differ from the hash recorded at the previous
renewal. `lease.ProgressSHA` carries that hash.
- **F17**, adapter, `f54fb00`. `pendingInput` scanned every line beginning with
the prompt marker, but queued and already-accepted input renders with the
same prefix. Only the editor owning the pane cursor is unsubmitted, so
`inputState` reads `#{cursor_y}` and captures screen rows without `-J`, which
would invalidate the row index. On that footing `ConfirmLaunch` became an
active submit protocol: resend Enter while the live editor still holds
exactly what was submitted, at most three times, no closer than two poll
intervals, then observe until the deadline. Queued input confirms rather than
fails. Evidence records `confirmation`, `submit_attempts` and both timestamps.
- **F19**, lifecycle, `edff021`. A task that reached the router's
`MaxAttempts` was permanently terminal. `TaskReleased` only increments
`Attempt`, `TaskCorrected` could not touch it, and no HTTP route emitted a
correction. `POST /v1/tasks/{id}/retry` requires the task to be failed,
unleased, and failed with `reason: retry_limit`, then appends one
`TaskCorrected` naming that failure with `state: queued` and `attempt: 0`.
Identity, goal, acceptance, decisions, phase and artifact refs all survive,
and the original failures stay in the log. `operation_id` is required and
makes it idempotent. Deliberately not a generic correction endpoint.
### The finding that mattered most
Run 3 failed three times with `prompt_not_submitted`, and six isolated probes
of the same code path could not reproduce it: fresh session, untrusted
directory with the trust dialog, `launch.md` present before startup, 100ms
readiness polling, `env -i` with only the worker's systemd variables, and the
real `StartAgent`/`Prompt`/`ConfirmLaunch` against a byte-identical git
worktree. All six submitted on the first Enter.
F17 turned that disagreement into data instead of a theory. The first live
launch after it deployed:
```
launch 06G44JZB80MZBEY97196EZN8EC confirmed: confirmation=editor_cleared
submit_attempts=2 first_submit_at=2026-08-27T09:12:00.801855937Z
confirmed_at=2026-08-27T09:12:01.315587139Z
```
Production needs the second Enter. Isolated probes need one. The submit is not
deterministic, which is exactly what F15's `ConfirmLaunch` comment asserted it
was.
### Still open
- **F18**, observability. A queued launch used to be misread as unsubmitted.
F17 fixes the predicate, but nothing else that parses pane text distinguishes
the active editor from history, so the same class of error can recur wherever
`PaneCapture` output is matched.
- **F9** is unchanged by F19. An operator still cannot release a lease someone
else owns without supplying that owner's `harness_id` and `lease_epoch`,
which means reading them out of the event log first. F19 recovers a terminal
task, which is a different escape hatch.
- **F6**, observability. Worker health reported `active_task: null` while the
store showed the task leased. Run 2 showed this was partly honest, because
the worker really had no working agent. Recheck now that launches confirm.
- **F4**, adapter quality. `harness "opencode" has no session-file resolver`,
so activity and thrash triggers are degraded on opencode.
- **F10**, hygiene. A coordinator-side release under a live worker leaves the
worker renewing a lease it no longer holds.
- Unset gated tokens: `ORCHESTRA_MCP_TOKEN`, `ORCHESTRA_MAVEN_TOKEN`,
`ORCHESTRA_AGENT_TOKEN`.
- `gofmt -l` fails on `internal/provider/provider.go`,
`internal/router/router.go` and `internal/webui/webui.go` at `edff021`. None
were touched by these runs. `internal/router/router.go` picked it up in
`0d67af9`. `go vet` passes, which is why nobody noticed.
### Operator actions taken by hand, and why
- Task `06G3ZCZWJ3QHF992ZMDGSJ0PYG`, run 2, was blocked with
`block_reason: operator_block` rather than released. A released task returns
to `queued`, and the router leases `queued` tasks, so releasing it would have
put it in competition with run 3.
- The first block attempt returned `task version conflict`.
`Store.validateTransition` fences every lifecycle event on a leased task, so
the payload needs `harness_id` and `lease_epoch` even though the HTTP handler
does not ask for them. That is F9 in practice.
- Task `06G44JZB80MZBEY97196EZN8EC`, run 3, was recovered with F19's retry
rather than by filing a fourth issue. That keeps `kami/test-e2e#3` served by
its own task and proves terminal recovery in the same run.
### Deployment boundary
Both halves report `edff021`, built from a detached worktree of that revision.
Worker sha256 `68ac265455acb0e007bfb0e898a91b317cd7289d7f2bf5da6df1224e1d6706a5`
at `/usr/local/bin/orchestra-worker`. Coordinator image built 13:08:30 +0400.