kami/orchestra
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Record runs 7 to 10 and the nine defects the plan burn-in found

Browse Source 
Twelve rungs proven, one not. The typed mismatch never fired, and both
attempts to force it tested the wrong thing: a file the agent can restore is
reversible, and a human correction is authoritative. Neither is a case where
the repository contradicts the plan.

Nine defects, all fixed. Three of them are one shape, a constraint the code
enforces and no brief states. The fourth is what made them expensive: a
refused handoff had no feedback loop, so the same rejected bytes were re-read
at every boundary until the task hit retry_limit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
This commit is contained in:
kami
2026-08-28 16:50:25 +04:00
parent 38aa0738a6
commit c4306dba28
1 changed files with 104 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BURNIN.md
+104
View File
@@ -1955,3 +1955,107 @@ machinery records into the same single `last_error` slot. That is the fourth,
fifth and sixth instance. The plan burn-in will produce exactly the five-second fifth and sixth instance. The plan burn-in will produce exactly the five-second
retry loops the bounded observation ring exists to make visible, so fix F18 retry loops the bounded observation ring exists to make visible, so fix F18
before running it. before running it.
## Runs 7 to 10, the plan machinery burn-in, 2026-08-28
Both halves were rebuilt and redeployed together seven times across these runs,
ending at `38aa073`. Every revision below was verified from the running
process, not the installed file.
`test-e2e` gained a verification policy in the coordinator's `config.jsonc`:
```json
"verification": { "allowed": [["bash", "-n", "*"], ["bash", "scripts/orchestra_e2e_healthcheck.sh"]] }
```
**The policy has one configuration copy, not two.** The worker never reads it:
`requestPlanVerification` calls `PlanPhaseCommands` on the coordinator and runs
only what comes back. A `verification` field in
`/etc/orchestra/worker-projects.json` would be dead configuration that later
reads as authoritative.
### What is proven
| Rung | Evidence |
|---|---|
| Research findings are citable | Run 10: seven findings, three confidence classes, five cited by the plan |
| The plan seals as a specification | Run 8: 7940 bytes, three phases, five `run:` lines, one manual step |
| The plan renders verbatim into implement | Run 8: the whole 7940-byte document byte for byte inside a 20822-byte `launch.md`, phase 3 heading and command present |
| Orchestra establishes phase progress | Run 8: `phase-1` verified at exit `[0]`, bound to `plan_ref`, `at_sha`, `evidence_ref`, harness and lease epoch |
| A manual step holds the phase | Runs 8, 9, 10: `awaiting_manual_verification` on every phase declaring one |
| A generic comment does not satisfy it | Run 8: "looks good to me" landed as `operator_instruction`, phase 2 unmoved |
| A keyed sign-off does | Run 8: `plan_phase_verification:51afbfa9…:phase-2`, phase 2 verified |
| Rotation preserves progress | Run 8: the successor received the whole plan and the progress block |
| Verification is bound to a SHA | Run 8: three phases rendered "verified at 14654d6cab63, stale because the tree is now at 56d9b9acdd59" |
| Reverification moves with the tree | Run 10: all three phases reverified at `ec0502f` after HEAD moved |
| A plan command outside policy is refused at seal | Run 10: refused on the planner, corrected in 25 seconds, then sealed |
| A legacy plan degrades honestly | Decoded from the real CAS artifact `1e12fcf0`: zero phases, 1646 bytes of markdown, the legacy notice present, the progress block empty |
| Review, resubmission and merge still work | Run 8: PR 11, changes requested, resubmitted, merged, receipt binds `ec320c82` |
### What is not proven
**The typed mismatch and the reopen.** `.orchestra/plan-mismatch.json` was never
written, across two deliberate attempts.
Both attempts tested the wrong thing, and the agent was right both times:
- Rewriting a file the plan depends on is *reversible*. The implementer
restored it from master and commented "Restore the byte-pinned USAGE
test_healthcheck.sh from master". The plan said the file gets no edit, an
edit appeared, and restoring it satisfied the plan.
- A human correction is *authoritative*. The brief tells the agent a decision
outranks the plan, so it followed the correction and recorded the stale plan
as an outstanding item: "The accepted plan text still specifies exit 2 and an
in-loop chain; it is stale against the correction."
`plan-mismatch.json` is for the case where the **repository** contradicts the
plan and no human has spoken. Proving it needs a purpose-built task where the
plan asserts something false about the repo that the implementer cannot make
true. The clean route is `POST /v1/tasks/<id>/phase` from the TUI surface,
which seals an authored artifact with no intervention race: seal a plan whose
phase says to extend a helper that does not exist.
### Defects found, all fixed
| ID | Commit | Defect |
|---|---|---|
| F47 | `4b32080` | Delivery merges the task branch, which carries `orchestra: TASK.md`. `writeTaskFile` returned early on `os.Stat`, so every later worktree inherited the previous task's file and every release failed "TASK.md changed". Run 7 died on `retry_limit` without leaving research. |
| F48 | `49409c9` | The manual plan-phase gate had two live consumers and no producer. Every imported comment was hardcoded to `operator_instruction`, so a phase with a manual step could never be verified. A comment whose first line is `orchestra verify <phase-id>` now carries the keyed subject. |
| F49 | `f4dbcf7` | A plan whose every phase was stale printed "Every phase is verified" under three lines that each said stale. |
| F50 | `4712c7d` | The research brief never stated the finding-id format the decoder enforces. Cost one boundary. |
| F51 | `98f1b2d`, `fda78cf` | The handoff parser split on `→` only and demanded the words "tried" and "failed because". Cost four leases across run 9 on content that was exactly right. |
| F52 | `fb7135e` | A plan command outside project policy sealed anyway. The only caller of `VerificationPolicy.Allows` was `PlanPhaseCommands`, which runs when the implementer asks to verify: one phase and one rotation too late. |
| F53 | `d7e75e9` | Only a verified phase consulted `Stale`. A pending manual gate rendered "automated checks passed at 94bd45c3b5d6" against a tree at `7d04aef`. |
| F54 | `015764e` | The handoff prompt stated the 200-character limit for `NEXT`, `WHY` and `REMAINING`. The validator applies it to `OPEN Q` and `LEARNED` too. |
| F55 | `38aa073` | **A refused handoff had no feedback loop.** `PrepareRelease` read the report, the parser refused it, the worker recorded it in health, and the next boundary read the same bytes. Run 10 spent four leases there. `answerRefusedHandoff` now says why and drops the file, gated on a typed `ErrInvalidHandoffAnswer` so transport failures keep their retry. |
F50, F51 and F54 are one shape: **a constraint the code enforces and no brief
states**. F55 is the shape that made them expensive, and it is the silent-loop
pattern `CLAUDE.md` names.
`fb7135e` has operational reach. Eight projects in `config.jsonc` carry no
`verification` block: `inference-router`, `Muzick`, `Kdrive`, `Nexus`,
`Praxis`, `Hexis`, `manga-pipeline`, `orchestra`. Their plans still seal, but
only with manual verification. A `run:` line is now refused on the planner.
### Recorded, not fixed
- **The operator lifecycle actions lose a version race.** `block`, `release`
and `attention` return `task version conflict` on a leased task, every
attempt, because the handler reads the version and the worker writes between
the read and the append. They succeed once the task is quiet.
- **`master` carries a stale `TASK.md`.** F47 makes it harmless, but every
pull request now shows a cosmetic `TASK.md` diff.
- **F18 is still open**, and it cost real diagnosis time here. The single
`last_error` slot held only the most recent failure, so the three earlier
ones in run 7 were overwritten before they could be read.
### Things that will bite
- The agent pane is on a private tmux socket:
`tmux -L orchestra capture-pane -p -t <session>:1.0`. A bare `tmux ls` does
not show it.
- A trivial task finishes three plan phases in about 105 seconds. Any
intervention timed against the implement phase will lose that race.
- `sudo` is unavailable in this sandbox, so every worker-side fix needs the
operator. Batch them: one install per run, not one per defect.