Reconcile docs with reality; fix module graph, token compare, health #1
@@ -486,3 +486,127 @@ F5 exposed done
|
||||
F8 fixed done
|
||||
fresh issue, no manual leasing
|
||||
```
|
||||
|
||||
## Runs 2 and 3, 2026-08-27
|
||||
|
||||
Written from live evidence. Neither run was a conformance pass, and both were
|
||||
worth more than one: run 2 found three blocking bugs behind each other, and
|
||||
run 3 found two more plus the reason six isolated probes disagreed with
|
||||
production.
|
||||
|
||||
`AUDIT.md` is uncommitted and owned by another session, so the burn-in ledger
|
||||
lives here.
|
||||
|
||||
### Fixed, with the revision each landed in
|
||||
|
||||
- **F12**, lifecycle, `0d67af9`. `Store.QuotaSince` reported an empty window as
|
||||
unknown, `QuotaAvailability` fails closed on unknown, and every herdr in
|
||||
`config.jsonc` declares a quota limit. The only producer of a receipt is a
|
||||
completed lease, so nothing could ever be leased. An empty window is now
|
||||
observable zero. A receipt that declares its own consumption unknown still
|
||||
fails closed. Verified live: run 2 leased three seconds after the fixed
|
||||
coordinator started.
|
||||
- **F13**, observability, `0d67af9`. `federatedAvailability` restated a quota
|
||||
refusal as worker health, so router health said `stale heartbeat` against a
|
||||
heartbeat one second old. Gates name themselves through
|
||||
`router.ReasonedAvailability`. Verified live.
|
||||
- **F14**, authority, `a5d361b`. Gitea ingest never set `acceptance`. One
|
||||
recognized heading, bullets and checkboxes until the next heading, order
|
||||
preserved, section removed from the description. Verified live on run 3: five
|
||||
ordered items, prose above the heading kept as the description.
|
||||
- **F15**, adapter, `1fd82f8`. Two bugs. `TmuxBackend.Prompt` wrote the whole
|
||||
instruction with `send-keys -l`, Claude Code coalesced it into a paste, and
|
||||
the Enter was absorbed. And `TaskLaunchAcknowledged` meant "Prompt returned
|
||||
nil", not "the harness accepted it". Launch transport became a backend
|
||||
property, and `ConfirmLaunch` began polling for proof. The detection works.
|
||||
The fix did not: see F17.
|
||||
- **F16**, lifecycle, `1888d42`. `renewLeases` renewed whenever a session
|
||||
existed and `PaneCapture` succeeded, so a pane that opened and never started
|
||||
renewed forever. This is why the July task stayed orphaned: F15 explains why
|
||||
nothing started, F16 why the lease never let go. Renewal now needs the agent
|
||||
busy, or the pane capture to differ from the hash recorded at the previous
|
||||
renewal. `lease.ProgressSHA` carries that hash.
|
||||
- **F17**, adapter, `f54fb00`. `pendingInput` scanned every line beginning with
|
||||
the prompt marker, but queued and already-accepted input renders with the
|
||||
same prefix. Only the editor owning the pane cursor is unsubmitted, so
|
||||
`inputState` reads `#{cursor_y}` and captures screen rows without `-J`, which
|
||||
would invalidate the row index. On that footing `ConfirmLaunch` became an
|
||||
active submit protocol: resend Enter while the live editor still holds
|
||||
exactly what was submitted, at most three times, no closer than two poll
|
||||
intervals, then observe until the deadline. Queued input confirms rather than
|
||||
fails. Evidence records `confirmation`, `submit_attempts` and both timestamps.
|
||||
- **F19**, lifecycle, `edff021`. A task that reached the router's
|
||||
`MaxAttempts` was permanently terminal. `TaskReleased` only increments
|
||||
`Attempt`, `TaskCorrected` could not touch it, and no HTTP route emitted a
|
||||
correction. `POST /v1/tasks/{id}/retry` requires the task to be failed,
|
||||
unleased, and failed with `reason: retry_limit`, then appends one
|
||||
`TaskCorrected` naming that failure with `state: queued` and `attempt: 0`.
|
||||
Identity, goal, acceptance, decisions, phase and artifact refs all survive,
|
||||
and the original failures stay in the log. `operation_id` is required and
|
||||
makes it idempotent. Deliberately not a generic correction endpoint.
|
||||
|
||||
### The finding that mattered most
|
||||
|
||||
Run 3 failed three times with `prompt_not_submitted`, and six isolated probes
|
||||
of the same code path could not reproduce it: fresh session, untrusted
|
||||
directory with the trust dialog, `launch.md` present before startup, 100ms
|
||||
readiness polling, `env -i` with only the worker's systemd variables, and the
|
||||
real `StartAgent`/`Prompt`/`ConfirmLaunch` against a byte-identical git
|
||||
worktree. All six submitted on the first Enter.
|
||||
|
||||
F17 turned that disagreement into data instead of a theory. The first live
|
||||
launch after it deployed:
|
||||
|
||||
```
|
||||
launch 06G44JZB80MZBEY97196EZN8EC confirmed: confirmation=editor_cleared
|
||||
submit_attempts=2 first_submit_at=2026-08-27T09:12:00.801855937Z
|
||||
confirmed_at=2026-08-27T09:12:01.315587139Z
|
||||
```
|
||||
|
||||
Production needs the second Enter. Isolated probes need one. The submit is not
|
||||
deterministic, which is exactly what F15's `ConfirmLaunch` comment asserted it
|
||||
was.
|
||||
|
||||
### Still open
|
||||
|
||||
- **F18**, observability. A queued launch used to be misread as unsubmitted.
|
||||
F17 fixes the predicate, but nothing else that parses pane text distinguishes
|
||||
the active editor from history, so the same class of error can recur wherever
|
||||
`PaneCapture` output is matched.
|
||||
- **F9** is unchanged by F19. An operator still cannot release a lease someone
|
||||
else owns without supplying that owner's `harness_id` and `lease_epoch`,
|
||||
which means reading them out of the event log first. F19 recovers a terminal
|
||||
task, which is a different escape hatch.
|
||||
- **F6**, observability. Worker health reported `active_task: null` while the
|
||||
store showed the task leased. Run 2 showed this was partly honest, because
|
||||
the worker really had no working agent. Recheck now that launches confirm.
|
||||
- **F4**, adapter quality. `harness "opencode" has no session-file resolver`,
|
||||
so activity and thrash triggers are degraded on opencode.
|
||||
- **F10**, hygiene. A coordinator-side release under a live worker leaves the
|
||||
worker renewing a lease it no longer holds.
|
||||
- Unset gated tokens: `ORCHESTRA_MCP_TOKEN`, `ORCHESTRA_MAVEN_TOKEN`,
|
||||
`ORCHESTRA_AGENT_TOKEN`.
|
||||
- `gofmt -l` fails on `internal/provider/provider.go`,
|
||||
`internal/router/router.go` and `internal/webui/webui.go` at `edff021`. None
|
||||
were touched by these runs. `internal/router/router.go` picked it up in
|
||||
`0d67af9`. `go vet` passes, which is why nobody noticed.
|
||||
|
||||
### Operator actions taken by hand, and why
|
||||
|
||||
- Task `06G3ZCZWJ3QHF992ZMDGSJ0PYG`, run 2, was blocked with
|
||||
`block_reason: operator_block` rather than released. A released task returns
|
||||
to `queued`, and the router leases `queued` tasks, so releasing it would have
|
||||
put it in competition with run 3.
|
||||
- The first block attempt returned `task version conflict`.
|
||||
`Store.validateTransition` fences every lifecycle event on a leased task, so
|
||||
the payload needs `harness_id` and `lease_epoch` even though the HTTP handler
|
||||
does not ask for them. That is F9 in practice.
|
||||
- Task `06G44JZB80MZBEY97196EZN8EC`, run 3, was recovered with F19's retry
|
||||
rather than by filing a fourth issue. That keeps `kami/test-e2e#3` served by
|
||||
its own task and proves terminal recovery in the same run.
|
||||
|
||||
### Deployment boundary
|
||||
|
||||
Both halves report `edff021`, built from a detached worktree of that revision.
|
||||
Worker sha256 `68ac265455acb0e007bfb0e898a91b317cd7289d7f2bf5da6df1224e1d6706a5`
|
||||
at `/usr/local/bin/orchestra-worker`. Coordinator image built 13:08:30 +0400.
|
||||
|
||||
Reference in New Issue
Block a user