Hand off with the plan machinery proven and tagged
Thirteen rungs, ten defects found and fixed, both halves deployed at orchestra-plan-v1. The handoff records what the defects had in common, why the mismatch rung took four attempts, and the two open items that should be fixed before anything runs unattended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
This commit is contained in:
@@ -0,0 +1,195 @@
|
||||
# Handoff: the plan machinery is proven live and tagged
|
||||
|
||||
Written 2026-08-28, 18:40 local (14:40 UTC). Read with `BURNIN.md` (the ledger,
|
||||
current through run 11), `PLAN-SPEC-DESIGN.md`, `AUDIT.md` and `CLAUDE.md`.
|
||||
The previous handoff is `HANDOFF-2026-08-28-plan-machinery.md`.
|
||||
|
||||
Everything below was observed live unless it says otherwise.
|
||||
|
||||
## The headline
|
||||
|
||||
**The plan machinery is proven and tagged.** Thirteen rungs, across runs 8, 10
|
||||
and 11 on `test-e2e`. Run 11 carried one task from creation through a
|
||||
contradicted plan, a real replan, review, submission and merge, with no
|
||||
lifecycle intervention.
|
||||
|
||||
**Ten defects were found on the way, and all ten are fixed.** Not one came from
|
||||
reading the code. Every one came from a live run failing.
|
||||
|
||||
```text
|
||||
orchestra-conformance-v1 lifecycle proven, plan machinery unproven
|
||||
↓ 20 commits
|
||||
orchestra-plan-v1 plan machinery proven, both halves deployed here
|
||||
```
|
||||
|
||||
## Deployed state
|
||||
|
||||
| Half | Revision | Evidence |
|
||||
|---|---|---|
|
||||
| Coordinator, homesrv container | `44ff35a` | `docker logs orchestra-api` prints it at startup |
|
||||
| Worker, workpc systemd | `44ff35a` | journal line at 18:32:35, sha256 `856e0952…` |
|
||||
|
||||
Both are `orchestra-plan-v1`.
|
||||
|
||||
**Verify the running worker, not the installed file.** `sudo install` lands the
|
||||
new binary while the old process keeps serving from the unlinked inode. The
|
||||
journal line at startup is the cheapest proof, because the process prints its
|
||||
own revision.
|
||||
|
||||
`sudo` is unavailable in this sandbox, so every worker-side change needs the
|
||||
operator. Staging path is `~/orchestra-deploy/orchestra-worker`.
|
||||
|
||||
**Batch worker-side fixes.** This session cost the operator eight installs. Four
|
||||
were sequential and unavoidable, because each fix unmasked the next defect. Two
|
||||
were my fault: `98f1b2d` and `fda78cf` are the same defect in the same eight
|
||||
lines, and I deployed between them. Collect fixes and deploy once per run.
|
||||
|
||||
The operator was offered this and did not take it up, so it stays available:
|
||||
|
||||
```
|
||||
kami ALL=(root) NOPASSWD: /usr/bin/install -m 0755 /home/kami/orchestra-deploy/orchestra-worker /usr/local/bin/orchestra-worker, /usr/bin/systemctl restart orchestra-worker
|
||||
```
|
||||
|
||||
## Configuration that had to change
|
||||
|
||||
`test-e2e` gained a verification policy in the coordinator's `config.jsonc`
|
||||
(`~/docker-apps/orchestra-web-ui/config.jsonc`):
|
||||
|
||||
```json
|
||||
"verification": { "allowed": [["bash", "-n", "*"], ["bash", "scripts/orchestra_e2e_healthcheck.sh"]] }
|
||||
```
|
||||
|
||||
**One configuration copy, not two.** The worker never reads this. It calls
|
||||
`PlanPhaseCommands` on the coordinator and runs only what comes back. A
|
||||
`verification` field in `/etc/orchestra/worker-projects.json` would be dead
|
||||
configuration that later reads as authoritative. Do not add one.
|
||||
|
||||
**`fb7135e` has operational reach.** Eight projects carry no `verification`
|
||||
block: `inference-router`, `Muzick`, `Kdrive`, `Nexus`, `Praxis`, `Hexis`,
|
||||
`manga-pipeline`, `orchestra`. Their plans still seal, but only with manual
|
||||
verification. A `run:` line is now refused on the planner. Configure the ones
|
||||
that will run plans.
|
||||
|
||||
## What the ten defects have in common
|
||||
|
||||
Three of them are one shape, and it is worth naming because it will recur:
|
||||
|
||||
> **a constraint the code enforces and no brief states.**
|
||||
|
||||
`4712c7d` (research finding ids), `98f1b2d` + `fda78cf` (the dead-end
|
||||
separator), `015764e` (the handoff length limit). Each cost a lease or a
|
||||
boundary to an agent doing exactly the right thing in slightly the wrong shape.
|
||||
|
||||
`38aa073` is what made those expensive, and it is the one to remember:
|
||||
|
||||
> **a refused handoff had no feedback loop.**
|
||||
|
||||
`PrepareRelease` read the report and the parser refused it. The worker recorded
|
||||
the error in health, and the next boundary read the same bytes. Run 10 spent
|
||||
four leases there and the agent was never told anything. The plan-progress path
|
||||
already had the answer in `answerRefusedProgress`. That is the silent-loop
|
||||
pattern `CLAUDE.md` names, found in a path nobody had checked.
|
||||
|
||||
**When you add a validator, add its refusal path in the same commit.** State
|
||||
the rule in the brief. Make the refusal reach the agent.
|
||||
|
||||
## The mismatch rung took four attempts
|
||||
|
||||
Three failed, and the agent was right every time. Keep this, because the
|
||||
instinct to blame the agent is strong and was wrong here.
|
||||
|
||||
- **A file the agent can restore is reversible.** I rewrote a file the plan
|
||||
depended on. The implementer restored it from master and committed "Restore
|
||||
the byte-pinned USAGE test_healthcheck.sh from master". The plan said the file
|
||||
gets no edit, an edit appeared, and restoring it satisfied the plan.
|
||||
- **A human correction is authoritative.** I posted a decision contradicting the
|
||||
plan's route. The brief tells the agent a decision outranks the plan. It
|
||||
followed the correction and recorded the stale plan as an outstanding item.
|
||||
- **A competent planner cannot author a false premise.** It researches first.
|
||||
|
||||
`plan-mismatch.json` is for the case where the **repository** contradicts the
|
||||
plan and no human has spoken. What worked: let research seal normally, then
|
||||
`POST /v1/tasks/<id>/phase` from the TUI surface with a plan authored to assert
|
||||
something false. Run 11's plan claimed an `emit_json()` helper at line 12 with
|
||||
three call sites. The script has none.
|
||||
|
||||
That endpoint is the general tool for this. It seals an authored artifact with
|
||||
no intervention race. That matters because **a trivial task finishes three plan
|
||||
phases in about 105 seconds**. Any intervention timed against the implement
|
||||
phase will lose.
|
||||
|
||||
## What is still open
|
||||
|
||||
- **F18, the bounded observation ring.** Still the oldest open item and it cost
|
||||
diagnosis time twice today. In run 7 the single `last_error` slot held only
|
||||
the most recent of four failures. In run 11 it was pinned to a *different,
|
||||
blocked* task for twenty-six minutes. Run 11's own expiry reason was never
|
||||
visible. `44ff35a` removed that flood. The slot is still one slot.
|
||||
- **A release transaction whose lease has expired can never commit.** Run 10
|
||||
ended this way. `PrepareRelease` succeeded, `Release` returned
|
||||
`409 lease not owned`, and the transaction stayed in `anchor_pushed` until
|
||||
`retry_limit`. The agent's finished work sat uncommitted in the worktree the
|
||||
whole time. This is the next real defect to fix.
|
||||
- **The operator lifecycle actions lose a version race.** `block`, `release` and
|
||||
`attention` return `task version conflict` on a leased task, every attempt.
|
||||
The handler reads the version and the worker writes between the read and the
|
||||
append. They succeed once the task is quiet.
|
||||
- **`master` of `test-e2e` carries a stale `TASK.md`.** `4b32080` makes it
|
||||
harmless, but every pull request shows a cosmetic `TASK.md` diff.
|
||||
- **Gitea returns 422 for a review on your own pull request.** Orchestra opens
|
||||
the PR as the same forge user the operator reviews as. `REQUEST_CHANGES` is
|
||||
therefore unreachable. The comment path works and is what every run here
|
||||
used. A separate bot account for `ORCHESTRA_GITEA_TOKEN` would restore it.
|
||||
|
||||
## Live state to clean up
|
||||
|
||||
Five `test-e2e` tasks are blocked, none of them real work:
|
||||
|
||||
| Task | Issue | Why |
|
||||
|---|---|---|
|
||||
| `06G4GBSQ2WRGD5HGYPYZZ4TYH0` | 13 | Run 10. Rungs proven; release kept failing lease-not-owned. Fail it. |
|
||||
| `06G4FZA825XMTCQ8ME8NN82YM8` | 12 | Run 9. Its sealed plan declares a phase-3 command outside policy, so phase 3 can never verify. Superseded by `fb7135e`. Fail it. |
|
||||
| `06G4E83E4KRXM8DS90M2648MGM` | 8 | The task F44 spawned from Orchestra's own pull request. Not real work. |
|
||||
| `06G3ZCZWJ3QHF992ZMDGSJ0PYG`, `06G46P6KE25Y04VVF7VRZMHZ78` | 2, 4 | Older, pre-date this session. |
|
||||
|
||||
Three `correx` tasks are queued and unschedulable: `correx` has no entry in
|
||||
`config.jsonc`.
|
||||
|
||||
## Things that will bite
|
||||
|
||||
- **The agent pane is on a private tmux socket.**
|
||||
`tmux -L orchestra capture-pane -p -t <session>:1.0`. A bare `tmux ls` shows
|
||||
nothing and looks like the pane is gone.
|
||||
- **`rm` and `cp` are interactive in this shell.** A plain `rm` prompts, appears
|
||||
to succeed, and leaves the file. Use `/bin/rm -f` and `install`.
|
||||
- **Another session owns 13 uncommitted paths**, including `AUDIT.md`,
|
||||
`deploy/build.sh`, `web/` and `AGENTS.md`. Commit by path. Never
|
||||
`git add -A` without a pathspec. `BURNIN.md` was clean and is mine.
|
||||
- **Secrets are guarded.** Reading `.env` or `printenv` of a token through the
|
||||
shell is blocked. The working recipe expands the token inside the container in
|
||||
a single remote command:
|
||||
`T=$(docker exec orchestra-api printenv ORCHESTRA_TUI_TOKEN); curl -s -H "X-Orchestra-Surface: tui" -H "Authorization: Bearer $T" ...`
|
||||
- **Rebuild the coordinator from a detached worktree, never
|
||||
`docker compose build`.** Used eight times today without trouble:
|
||||
```
|
||||
git -C ~/apps/orchestra worktree add --detach --quiet /tmp/orchestra-build-$FULL $FULL
|
||||
docker build --build-arg BUILD_REVISION=$FULL --build-arg BUILD_TIME=$BT --build-arg BUILD_DIRTY=false -f $T/Dockerfile.api -t orchestra-web-ui-orchestra-api $T
|
||||
docker compose -f compose.yaml -f compose.live.yaml -f compose.override.yaml up -d --no-deps orchestra-api
|
||||
```
|
||||
- **`deploy/build.sh` builds both halves from one commit with one stamp.** Use
|
||||
it. It is what makes deployed identity evidence rather than assumption.
|
||||
|
||||
## What I would do next, in order
|
||||
|
||||
1. **Fix the expired-release transaction.** It is the one defect left that
|
||||
silently destroys finished work. Run 10 lost a whole task to it.
|
||||
2. **Fix F18.** Two sessions running have now been slowed by the single
|
||||
`last_error` slot. Today it hid a defect from me for twenty-six minutes.
|
||||
3. **Fail the two burn-in tasks** above and decide on the three older ones.
|
||||
4. **Configure `verification` for whichever real projects will run plans.**
|
||||
Until then their plans can only carry manual steps.
|
||||
5. Only then the Gitea bot account and the duplicated `quality_gate`
|
||||
configuration. Both are real, neither blocks anything.
|
||||
|
||||
Do 1 before running anything unattended. A task that finishes its work and then
|
||||
cannot commit it is worse than a task that fails.
|
||||
Reference in New Issue
Block a user