Hand off mid-observation, with the evidence standard
Run 3 is waiting on an autonomous expiry and relaunch, and the next session has to judge it without repeating this one mistake: F16 was reported as passing live on the strength of renewal events, while its predicate was inert. The hierarchy the operator set is at the top of the file, and the two hard failure predicates are stated before anything else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,167 @@
|
||||
# Handoff — run 3 relaunch pending, two hard predicates open
|
||||
|
||||
Written 2026-08-27, 11:00 UTC, mid-observation. Read with
|
||||
`HANDOFF-2026-08-27-burnin.md` (the session before this one), `BURNIN.md`
|
||||
(the ledger, now current through run 3), `AUDIT.md` and `CLAUDE.md`.
|
||||
|
||||
Everything below was observed live unless it says otherwise.
|
||||
|
||||
## Evidence standard, set by the operator
|
||||
|
||||
Use this hierarchy and do not mix the tiers:
|
||||
|
||||
```
|
||||
event log → lifecycle truth
|
||||
worker journal → worker-local observations and confirmer receipts
|
||||
pane capture → harness evidence only
|
||||
code-path argument → supporting evidence, not live proof
|
||||
```
|
||||
|
||||
This exists because this session reported "F16 passes its first branch live"
|
||||
on the strength of three `TaskLeaseRenewed` events. The predicate was inert.
|
||||
A code-path argument was presented as live proof, and the bug survived the
|
||||
report. Do not repeat that.
|
||||
|
||||
## Deployed state
|
||||
|
||||
| Half | Revision | Evidence |
|
||||
|---|---|---|
|
||||
| Coordinator, homesrv container | `edbe98f` | `docker logs orchestra-api`, 10:45:46 UTC |
|
||||
| Worker, workpc systemd | `edbe98f` | `journalctl -u orchestra-worker`, 14:45:26 +04 |
|
||||
|
||||
Branch `webui-and-audit-reconciliation`, HEAD `edbe98f`. Worker sha256
|
||||
`e94ab4ddbbd755796ca68e273a55029895ec930ea88405be2f6af65be9bdc494`, staged at
|
||||
`~/orchestra-deploy/orchestra-worker.edbe98f`.
|
||||
|
||||
Both halves match. Do not start a new run until they do again after any change.
|
||||
|
||||
## What is being watched right now
|
||||
|
||||
Task `06G44JZB80MZBEY97196EZN8EC`, issue `kami/test-e2e#3`, is `leased` at v15
|
||||
under **stranded epoch `06G44ZBZX4YH6PRN7Y4ZH8GG3W`**. Its pane is gone and its
|
||||
agent has been idle since 09:13. The last renewal was seq 358 at 10:32:09, by
|
||||
the old worker. The lease runs to roughly 11:02 UTC.
|
||||
|
||||
Nobody should touch it. The expiry is the test.
|
||||
|
||||
Expected chain, autonomous, no operator action:
|
||||
|
||||
```
|
||||
renewal boundary reached
|
||||
→ progress observation fails, pane is gone
|
||||
→ worker does not renew
|
||||
→ lease expires
|
||||
→ TaskReleased(reason=lease_expired)
|
||||
→ queued, attempt 1
|
||||
→ router selects the worker
|
||||
→ new lease, new epoch, new pane
|
||||
→ confirmed input submission
|
||||
→ TaskLaunchAcknowledged
|
||||
```
|
||||
|
||||
### Two hard failure predicates
|
||||
|
||||
```
|
||||
old epoch 06G44ZBZX4YH6PRN7Y4ZH8GG3W renewed after 10:45:26 → F16 fail
|
||||
first orchestra-originated post-frame input has no confirmer receipt → F20 fail
|
||||
```
|
||||
|
||||
The F20 receipt is strictly this line in the worker journal:
|
||||
|
||||
```
|
||||
input to <pane> confirmed: confirmation=<kind> submit_attempts=<n> ...
|
||||
```
|
||||
|
||||
Pane movement, text leaving the editor, and later model output are all
|
||||
insufficient.
|
||||
|
||||
### Checks to capture, in order
|
||||
|
||||
1. No `TaskLeaseRenewed` carrying the stranded epoch after 10:45:26.
|
||||
2. Release reason is literally `lease_expired`, not a restart or reconcile
|
||||
fallback.
|
||||
3. New lease epoch differs from `06G44ZBZX4YH6PRN7Y4ZH8GG3W`.
|
||||
4. `ProgressSHA`, `UsageBaseline` and `PickupAcknowledged` start fresh for the
|
||||
new epoch. State this precisely: the reset is **structurally established by
|
||||
the reconciler**, and the first successful renewal under the new epoch shows
|
||||
the reconstructed lease stayed usable. Those fields are not exposed, so do
|
||||
not claim their runtime values were observed.
|
||||
5. Launch confirmation evidence carries `confirmation=` and `submit_attempts=`.
|
||||
6. No operator action between expiry and relaunch.
|
||||
7. Then the F20 boundary: the first Orchestra-originated input after `frame`
|
||||
completes.
|
||||
|
||||
If the task reaches real implementation after that with no manual
|
||||
intervention, this stops being a diagnostic run.
|
||||
|
||||
## Fixed since the last handoff
|
||||
|
||||
Each landed in the revision named. `BURNIN.md` carries the full entries.
|
||||
|
||||
- **F16**, lifecycle, `1888d42` then `770cc6a` then `edbe98f`. Renewal requires
|
||||
observable progress. It shipped **inert**: `reconcileLeases` rebuilt every
|
||||
lease from the coordinator's task list each tick and dropped `ProgressSHA`,
|
||||
so the check took its no-baseline branch forever. `UsageBaseline` and
|
||||
`PickupAcknowledged` were wiped the same way. Progress is now measured with
|
||||
input lines removed, so keystrokes are not work.
|
||||
- **F17**, adapter, `f54fb00`. `pendingInput` scanned every line with the
|
||||
prompt marker, but queued and accepted input renders identically. Only the
|
||||
editor owning the cursor is unsubmitted. `ConfirmLaunch` became an active
|
||||
submit protocol: resend Enter while the live editor holds exactly what was
|
||||
submitted, at most 3 times, then observe. Verified live:
|
||||
`submit_attempts=2` in production against 1 in six isolated probes.
|
||||
- **F19**, lifecycle, `edff021`. `POST /v1/tasks/{id}/retry`, full-control
|
||||
surfaces only, for a failed and unleased task whose failure reason is
|
||||
`retry_limit`. Requires `operation_id` and is idempotent. Verified live: the
|
||||
task leased itself one second after the correction.
|
||||
- **F20**, adapter, `edbe98f`. Every Orchestra-originated pane write confirms.
|
||||
`LaunchConfirmer` is `InputConfirmer`, `ConfirmLaunch` is `ConfirmInput`, and
|
||||
`sendPrompt` and `sendLine` both route through it. Tests only so far.
|
||||
- **F12**, **F13**, **F14** landed earlier and are described in `BURNIN.md`.
|
||||
F12 and F13 were verified live, F14 on run 3's ingest.
|
||||
|
||||
## Open
|
||||
|
||||
- **F20 live**, unproven. This is the next boundary.
|
||||
- **F18**, observability. Nothing besides `inputState` distinguishes the active
|
||||
editor from history when parsing pane text, so the same class of error can
|
||||
recur wherever `PaneCapture` output is matched.
|
||||
- **F9**, unchanged. An operator cannot release a lease someone else owns
|
||||
without supplying that owner's `harness_id` and `lease_epoch`, read out of
|
||||
the event log first. F19 did not address this.
|
||||
- **Approval input is unconfirmed on purpose.** `main.go:1078` writes `y`, `n`
|
||||
and `ENTER` to a y/n dialog, not an editor, so the editor predicate would
|
||||
report nonsense. It needs its own predicate if it is ever covered.
|
||||
- **F6**, observability. Worker health reported `active_task: null` against a
|
||||
leased task. Recheck now that launches confirm.
|
||||
- **F4**, adapter quality. `harness "opencode" has no session-file resolver`.
|
||||
- **F10**, hygiene. A coordinator-side release under a live worker leaves the
|
||||
worker renewing a lease it no longer holds.
|
||||
- Unset gated tokens: `ORCHESTRA_MCP_TOKEN`, `ORCHESTRA_MAVEN_TOKEN`,
|
||||
`ORCHESTRA_AGENT_TOKEN`.
|
||||
- `gofmt -l` fails on `internal/provider/provider.go`,
|
||||
`internal/router/router.go` and `internal/webui/webui.go`. Untouched by these
|
||||
runs, and `go vet` passes, which is why nobody noticed.
|
||||
|
||||
## Things that will bite
|
||||
|
||||
- **Another session still owns 18 uncommitted paths**, including `AUDIT.md`.
|
||||
Commit by path. Never `git add -A`. The burn-in ledger lives in `BURNIN.md`
|
||||
for exactly this reason.
|
||||
- **An unexplained line was typed into the run-3 pane.** The operator confirms
|
||||
it was not them, and the worker journal shows no send. Treat unexplained pane
|
||||
input as possible, which is why progress must ignore input entirely.
|
||||
- **Do not use `docker compose build`.** Its context is the shared live
|
||||
checkout. Build from a detached worktree, as `deploy/build.sh` does and as
|
||||
the coordinator image command in `HANDOFF-2026-08-26-burnin.md` shows.
|
||||
- **Installing the worker needs root**, so it is always the operator's step.
|
||||
Mutations through the API need the TUI token from the compose `.env`, which
|
||||
`secret-guard` blocks from being read. Hand the operator the command.
|
||||
- **This machine is workpc**, `hostname` is `bugmachine`. homesrv is
|
||||
`kami@192.168.1.104` with `/usr/bin/ssh`, not the `ssh` on PATH.
|
||||
- **`/tmp` is tmpfs with a 10-day sweep.** Check `/tmp/test-e2e` and
|
||||
`/tmp/test-e2e-worktrees` exist before every run.
|
||||
- **The MCP surface is read-only and needs no credential.**
|
||||
`curl -H 'X-Orchestra-Surface: mcp' http://192.168.1.104:9145/v1/tasks`.
|
||||
- **Three queued `correx` tasks will never lease.** No worker declares that
|
||||
project. Router health says so.
|
||||
Reference in New Issue
Block a user