kami/orchestra
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Record runs 7 to 10 and the nine defects the plan burn-in found

Browse Source 
Twelve rungs proven, one not. The typed mismatch never fired, and both
attempts to force it tested the wrong thing: a file the agent can restore is
reversible, and a human correction is authoritative. Neither is a case where
the repository contradicts the plan.

Nine defects, all fixed. Three of them are one shape, a constraint the code
enforces and no brief states. The fourth is what made them expensive: a
refused handoff had no feedback loop, so the same rejected bytes were re-read
at every boundary until the task hit retry_limit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
This commit is contained in:
kami
2026-08-28 16:50:25 +04:00
parent 38aa0738a6
commit c4306dba28
1 changed files with 104 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BURNIN.md
+104
View File
@@ -1955,3 +1955,107 @@ machinery records into the same single `last_error` slot. That is the fourth,
fifth and sixth instance. The plan burn-in will produce exactly the five-second
retry loops the bounded observation ring exists to make visible, so fix F18
before running it.
## Runs 7 to 10, the plan machinery burn-in, 2026-08-28
Both halves were rebuilt and redeployed together seven times across these runs,
ending at `38aa073`. Every revision below was verified from the running
process, not the installed file.
`test-e2e` gained a verification policy in the coordinator's `config.jsonc`:
```json
"verification": { "allowed": [["bash", "-n", "*"], ["bash", "scripts/orchestra_e2e_healthcheck.sh"]] }
```
**The policy has one configuration copy, not two.** The worker never reads it:
`requestPlanVerification` calls `PlanPhaseCommands` on the coordinator and runs
only what comes back. A `verification` field in
`/etc/orchestra/worker-projects.json` would be dead configuration that later
reads as authoritative.
### What is proven
| Rung | Evidence |
|---|---|
| Research findings are citable | Run 10: seven findings, three confidence classes, five cited by the plan |
| The plan seals as a specification | Run 8: 7940 bytes, three phases, five `run:` lines, one manual step |
| The plan renders verbatim into implement | Run 8: the whole 7940-byte document byte for byte inside a 20822-byte `launch.md`, phase 3 heading and command present |
| Orchestra establishes phase progress | Run 8: `phase-1` verified at exit `[0]`, bound to `plan_ref`, `at_sha`, `evidence_ref`, harness and lease epoch |
| A manual step holds the phase | Runs 8, 9, 10: `awaiting_manual_verification` on every phase declaring one |
| A generic comment does not satisfy it | Run 8: "looks good to me" landed as `operator_instruction`, phase 2 unmoved |
| A keyed sign-off does | Run 8: `plan_phase_verification:51afbfa9…:phase-2`, phase 2 verified |
| Rotation preserves progress | Run 8: the successor received the whole plan and the progress block |
| Verification is bound to a SHA | Run 8: three phases rendered "verified at 14654d6cab63, stale because the tree is now at 56d9b9acdd59" |
| Reverification moves with the tree | Run 10: all three phases reverified at `ec0502f` after HEAD moved |
| A plan command outside policy is refused at seal | Run 10: refused on the planner, corrected in 25 seconds, then sealed |
| A legacy plan degrades honestly | Decoded from the real CAS artifact `1e12fcf0`: zero phases, 1646 bytes of markdown, the legacy notice present, the progress block empty |
| Review, resubmission and merge still work | Run 8: PR 11, changes requested, resubmitted, merged, receipt binds `ec320c82` |
### What is not proven
**The typed mismatch and the reopen.** `.orchestra/plan-mismatch.json` was never
written, across two deliberate attempts.
Both attempts tested the wrong thing, and the agent was right both times:
- Rewriting a file the plan depends on is *reversible*. The implementer
restored it from master and commented "Restore the byte-pinned USAGE
test_healthcheck.sh from master". The plan said the file gets no edit, an
edit appeared, and restoring it satisfied the plan.
- A human correction is *authoritative*. The brief tells the agent a decision
outranks the plan, so it followed the correction and recorded the stale plan
as an outstanding item: "The accepted plan text still specifies exit 2 and an
in-loop chain; it is stale against the correction."
`plan-mismatch.json` is for the case where the **repository** contradicts the
plan and no human has spoken. Proving it needs a purpose-built task where the
plan asserts something false about the repo that the implementer cannot make
true. The clean route is `POST /v1/tasks/<id>/phase` from the TUI surface,
which seals an authored artifact with no intervention race: seal a plan whose
phase says to extend a helper that does not exist.
### Defects found, all fixed
| ID | Commit | Defect |
|---|---|---|
| F47 | `4b32080` | Delivery merges the task branch, which carries `orchestra: TASK.md`. `writeTaskFile` returned early on `os.Stat`, so every later worktree inherited the previous task's file and every release failed "TASK.md changed". Run 7 died on `retry_limit` without leaving research. |
| F48 | `49409c9` | The manual plan-phase gate had two live consumers and no producer. Every imported comment was hardcoded to `operator_instruction`, so a phase with a manual step could never be verified. A comment whose first line is `orchestra verify <phase-id>` now carries the keyed subject. |
| F49 | `f4dbcf7` | A plan whose every phase was stale printed "Every phase is verified" under three lines that each said stale. |
| F50 | `4712c7d` | The research brief never stated the finding-id format the decoder enforces. Cost one boundary. |
| F51 | `98f1b2d`, `fda78cf` | The handoff parser split on `→` only and demanded the words "tried" and "failed because". Cost four leases across run 9 on content that was exactly right. |
| F52 | `fb7135e` | A plan command outside project policy sealed anyway. The only caller of `VerificationPolicy.Allows` was `PlanPhaseCommands`, which runs when the implementer asks to verify: one phase and one rotation too late. |
| F53 | `d7e75e9` | Only a verified phase consulted `Stale`. A pending manual gate rendered "automated checks passed at 94bd45c3b5d6" against a tree at `7d04aef`. |
| F54 | `015764e` | The handoff prompt stated the 200-character limit for `NEXT`, `WHY` and `REMAINING`. The validator applies it to `OPEN Q` and `LEARNED` too. |
| F55 | `38aa073` | **A refused handoff had no feedback loop.** `PrepareRelease` read the report, the parser refused it, the worker recorded it in health, and the next boundary read the same bytes. Run 10 spent four leases there. `answerRefusedHandoff` now says why and drops the file, gated on a typed `ErrInvalidHandoffAnswer` so transport failures keep their retry. |
F50, F51 and F54 are one shape: **a constraint the code enforces and no brief
states**. F55 is the shape that made them expensive, and it is the silent-loop
pattern `CLAUDE.md` names.
`fb7135e` has operational reach. Eight projects in `config.jsonc` carry no
`verification` block: `inference-router`, `Muzick`, `Kdrive`, `Nexus`,
`Praxis`, `Hexis`, `manga-pipeline`, `orchestra`. Their plans still seal, but
only with manual verification. A `run:` line is now refused on the planner.
### Recorded, not fixed
- **The operator lifecycle actions lose a version race.** `block`, `release`
and `attention` return `task version conflict` on a leased task, every
attempt, because the handler reads the version and the worker writes between
the read and the append. They succeed once the task is quiet.
- **`master` carries a stale `TASK.md`.** F47 makes it harmless, but every
pull request now shows a cosmetic `TASK.md` diff.
- **F18 is still open**, and it cost real diagnosis time here. The single
`last_error` slot held only the most recent failure, so the three earlier
ones in run 7 were overwritten before they could be read.
### Things that will bite
- The agent pane is on a private tmux socket:
`tmux -L orchestra capture-pane -p -t <session>:1.0`. A bare `tmux ls` does
not show it.
- A trivial task finishes three plan phases in about 105 seconds. Any
intervention timed against the implement phase will lose that race.
- `sudo` is unavailable in this sandbox, so every worker-side fix needs the
operator. Batch them: one install per run, not one per defect.