Hand off the burn-in: run 2 found three bugs, run 3 needs an install
Run 2 never reached conformance. It found F12, then F15, then F16, each
hidden behind the one before it. F12, F13, F14 and F15 are fixed and
committed; F16 is diagnosed and open. Neither deployed half is at 1fd82f8
yet, and the worker install needs root, so the next session starts with an
operator step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,159 @@
|
||||
# Handoff — burn-in, run 2 diagnosed, run 3 blocked on an operator step
|
||||
|
||||
Written 2026-08-27, early. Read with `HANDOFF-2026-08-26-burnin.md` (the
|
||||
previous session), `BURNIN.md`, `AUDIT.md` and `CLAUDE.md`. Everything below
|
||||
was observed live. Nothing here is inferred from a passing test.
|
||||
|
||||
Run 2 was not a conformance run. It was worth more than one: it found three
|
||||
blocking bugs in a row, each hidden behind the previous one.
|
||||
|
||||
## Deployed state, and the mismatch
|
||||
|
||||
| Half | Live revision | Built and staged | Evidence |
|
||||
|---|---|---|---|
|
||||
| Coordinator, homesrv container | `0d67af9` | image for `1fd82f8` built, not recreated | `docker logs orchestra-api` startup line |
|
||||
| Worker, workpc systemd | `77a2b32` | `~/orchestra-deploy/orchestra-worker.1fd82f8` | `journalctl -u orchestra-worker` |
|
||||
|
||||
Branch `webui-and-audit-reconciliation`, HEAD `1fd82f8`.
|
||||
|
||||
Staged worker sha256 `19e17893d6f87651050c9826e8264aba313d2ed4385013efd4576e13d5b7c9ab`.
|
||||
The installed `/usr/local/bin/orchestra-worker` is `2a850f2d1102d390fe5960011756e14cd14ed14f6e15384189e63e961eb419b1`.
|
||||
|
||||
Neither half is at `1fd82f8`. Run 3 cannot start until both are.
|
||||
|
||||
## Do this first
|
||||
|
||||
Installing the worker needs root, so the sequence starts with the operator.
|
||||
It must run in this order, and the reason is F16 below.
|
||||
|
||||
1. Operator stops `orchestra-worker`. This is what stops the stale renewal.
|
||||
2. Kill only the run 2 tmux session:
|
||||
`tmux -L orchestra kill-session -t orchestra-06g3zczwj3qhf992zmdgsj0pyg-6b87ef68`.
|
||||
Nothing else on that socket. Other sessions there are not Orchestra's.
|
||||
3. Operator installs `~/orchestra-deploy/orchestra-worker.1fd82f8` to
|
||||
`/usr/local/bin/orchestra-worker` and starts the unit. Verify the startup
|
||||
line reads revision `1fd82f8`.
|
||||
4. Recreate the coordinator. The image is already built:
|
||||
`cd /home/kami/docker-apps/orchestra-web-ui && docker compose -f compose.yaml -f compose.live.yaml up -d --no-deps --force-recreate orchestra-api`
|
||||
5. Verify both revisions and a fresh heartbeat at
|
||||
`GET /v1/federation/workers` before creating anything.
|
||||
6. Decide what happens to task `06G3ZCZWJ3QHF992ZMDGSJ0PYG` (see below), then
|
||||
create the run 3 issue.
|
||||
|
||||
## The open decision
|
||||
|
||||
Task `06G3ZCZWJ3QHF992ZMDGSJ0PYG` (issue `kami/test-e2e#2`) is still leased to
|
||||
epoch `06G3ZGYSFGWKSV9CHJ4EADQVBC` and is at version 5. Once the worker stops
|
||||
renewing it, the lease expires and the router will re-lease it under the fixed
|
||||
worker. That would be a genuine F15 test. It also competes with run 3 for a
|
||||
single-capacity worker, and it is not a fresh issue.
|
||||
|
||||
The operator was asked and had not answered when this session ended. Ask again
|
||||
before creating run 3.
|
||||
|
||||
## What run 2 proved
|
||||
|
||||
Autonomous assignment works. The lease landed three seconds after the fixed
|
||||
coordinator started, with no manual intervention:
|
||||
|
||||
```
|
||||
306 TaskLeased 20:29:54 workpc-claude epoch 06G3ZGYSFGWKSV9CHJ4EADQVBC
|
||||
307 TaskLaunchAcknowledged 20:30:00 pane orchestra-06g3zczwj3qhf992zmdgsj0pyg-6b87ef68:1.0
|
||||
308 TaskLeaseRenewed 20:49:59 same epoch
|
||||
309 TaskLeaseRenewed 21:10:04 same epoch
|
||||
```
|
||||
|
||||
`launch.md` carried a body-derived goal, the authority order, the ambiguity
|
||||
ladder, the `frame` phase brief and a verified git state.
|
||||
|
||||
And then nothing ran. Events 308 and 309 are a lease renewing itself around an
|
||||
agent that never started.
|
||||
|
||||
## Fixed this session
|
||||
|
||||
- **F12**, lifecycle, was blocking every autonomous lease. `Store.QuotaSince`
|
||||
reported an empty window as unknown, `QuotaAvailability` fails closed on
|
||||
unknown, and every herdr in `config.jsonc` declares a quota limit. The only
|
||||
producer of a receipt is a completed lease, so nothing could ever be leased.
|
||||
The live event log had zero `QuotaReported` events. An empty window is now
|
||||
observable zero. A receipt that declares its own consumption unknown still
|
||||
fails closed. `0d67af9`.
|
||||
- **F13**, observability. `federatedAvailability` restated a quota refusal as
|
||||
worker health, so router health said `stale heartbeat` against a heartbeat
|
||||
one second old. Gates now name themselves through `router.ReasonedAvailability`.
|
||||
`0d67af9`.
|
||||
- **F14**, authority. Gitea ingest never set `acceptance`. One recognized
|
||||
heading (`acceptance` or `acceptance criteria`), bullets and checkboxes until
|
||||
the next heading, order preserved, section removed from the description. No
|
||||
section means no acceptance, which is correct rather than a failure.
|
||||
`a5d361b`.
|
||||
- **F15**, adapter and lifecycle, found live. Two bugs. `TmuxBackend.Prompt`
|
||||
wrote the whole instruction with `send-keys -l`, Claude Code coalesced it
|
||||
into a paste, and the following Enter was absorbed. And
|
||||
`TaskLaunchAcknowledged` meant "Prompt returned nil", not "the harness
|
||||
accepted it". Launch transport is now a backend property: claude on tmux
|
||||
submits one line pointing at `.orchestra/launch.md`, everything else stays
|
||||
inline. `ConfirmLaunch` polls until the input editor clears and the agent is
|
||||
observably busy or blocked. An editor still holding the prompt at the
|
||||
deadline returns `herdr.ErrPromptNotSubmitted`, the worker kills the pane and
|
||||
drops the session, and `classifyLaunchError` returns `prompt_not_submitted`
|
||||
instead of `launch_uncertain`. `1fd82f8`.
|
||||
|
||||
`go build ./...`, `go vet ./...` and `go test ./...` all pass at `1fd82f8`.
|
||||
F12 and F13 were verified live. F14 and F15 have tests only.
|
||||
|
||||
## Open findings, in the order this session would take them
|
||||
|
||||
- **F16**, lifecycle, new and unfixed. `renewLeases`
|
||||
(`cmd/orchestra-worker/main.go:951`) renews whenever a session exists and
|
||||
`PaneCapture` succeeds. Nothing requires the agent to have accepted a prompt
|
||||
or done any work, so a pane that opened and never started renews its lease
|
||||
forever. This is the mechanism behind the July stuck task: F15 explains why
|
||||
nothing started, F16 explains why the lease never let go. A fix should
|
||||
require observable progress, not pane existence. F15's `ConfirmLaunch` gives
|
||||
the shape to reuse.
|
||||
- **F6**, observability. Worker health reports `active_task: null` while the
|
||||
store shows the task leased. Run 2 showed this was partly honest: the worker
|
||||
really had no working agent. Recheck after F15 is live before treating it as
|
||||
a reporting bug.
|
||||
- **F4**, adapter quality. `harness "opencode" has no session-file resolver`,
|
||||
so activity and thrash triggers are degraded on opencode.
|
||||
- **F9**, lifecycle. Releasing a leased task through
|
||||
`POST /v1/tasks/<id>/release` requires `harness_id` and `lease_epoch`. There
|
||||
is no operator escape hatch. Run 2 made this concrete: with F16 renewing
|
||||
forever, there was no way to end the run without stopping the worker.
|
||||
- **F10**, hygiene. A coordinator-side release under a live worker leaves the
|
||||
worker renewing a lease it no longer holds.
|
||||
- **Unset gated tokens**: `ORCHESTRA_MCP_TOKEN`, `ORCHESTRA_MAVEN_TOKEN` and
|
||||
`ORCHESTRA_AGENT_TOKEN` are still empty.
|
||||
|
||||
## Things that will bite
|
||||
|
||||
- **Another session still owns 18 uncommitted paths.** `internal/authn/*`,
|
||||
`web/*`, `internal/webui/assets/*`, `AGENTS.md`, `deploy/DEPLOYMENT.md`,
|
||||
`deploy/orchestra.env.example`, `AUDIT.md`, and the `deploy/build.sh`
|
||||
extension that also builds `cmd/orchestra-user`. Commit by path. Never
|
||||
`git add -A`. This session did not touch them, including `AUDIT.md`, which
|
||||
is why the findings above are not in the ledger yet.
|
||||
- **`deploy/build.sh` is one of those uncommitted files.** It works and this
|
||||
session used it, but it is not in any commit.
|
||||
- **Do not use `docker compose build`.** Its context is the shared live
|
||||
checkout. Build from a detached worktree of the revision, as in
|
||||
`HANDOFF-2026-08-26-burnin.md`.
|
||||
- **This machine is workpc.** `hostname` is `bugmachine`. The worker, the
|
||||
tmux socket `orchestra`, `/tmp/test-e2e` and `/tmp/test-e2e-worktrees` are
|
||||
all local here. There is no SSH key from here to `192.168.1.105`, and none
|
||||
is needed. homesrv is reachable at `kami@192.168.1.104` with `/usr/bin/ssh`;
|
||||
the `ssh` on PATH is kitty's interactive-only wrapper and will fail.
|
||||
- **`/tmp/test-e2e` survived**, head `faba139`, non-bare clone. `/tmp` is
|
||||
tmpfs with a 10-day sweep, so check it exists before every run.
|
||||
- **The real script is `scripts/orchestra_e2e_healthcheck.sh`**, 17 lines,
|
||||
one `--help` flag, no per-check structure. Write issue bodies against what
|
||||
is actually in the repo.
|
||||
- **The MCP surface is a read-only diagnostic with no credential.**
|
||||
`curl -H 'X-Orchestra-Surface: mcp' http://192.168.1.104:9145/v1/tasks` is
|
||||
how this session read live state. Mutations need the TUI token from `.env`.
|
||||
- **`secret-guard` blocks reading `.env`.** Ask the operator for the one value
|
||||
needed, or have them run the command.
|
||||
- **The three queued `correx` tasks will never lease.** No worker declares
|
||||
that project. Router health says so.
|
||||
Reference in New Issue
Block a user