From c4306dba288c3fb63e78c8e7c765861bf4d52a25 Mon Sep 17 00:00:00 2001 From: kami Date: Fri, 28 Aug 2026 16:50:25 +0400 Subject: [PATCH] Record runs 7 to 10 and the nine defects the plan burn-in found Twelve rungs proven, one not. The typed mismatch never fired, and both attempts to force it tested the wrong thing: a file the agent can restore is reversible, and a human correction is authoritative. Neither is a case where the repository contradicts the plan. Nine defects, all fixed. Three of them are one shape, a constraint the code enforces and no brief states. The fourth is what made them expensive: a refused handoff had no feedback loop, so the same rejected bytes were re-read at every boundary until the task hit retry_limit. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1 --- BURNIN.md | 104 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) diff --git a/BURNIN.md b/BURNIN.md index 88792fd..d4416fb 100644 --- a/BURNIN.md +++ b/BURNIN.md @@ -1955,3 +1955,107 @@ machinery records into the same single `last_error` slot. That is the fourth, fifth and sixth instance. The plan burn-in will produce exactly the five-second retry loops the bounded observation ring exists to make visible, so fix F18 before running it. + +## Runs 7 to 10, the plan machinery burn-in, 2026-08-28 + +Both halves were rebuilt and redeployed together seven times across these runs, +ending at `38aa073`. Every revision below was verified from the running +process, not the installed file. + +`test-e2e` gained a verification policy in the coordinator's `config.jsonc`: + +```json +"verification": { "allowed": [["bash", "-n", "*"], ["bash", "scripts/orchestra_e2e_healthcheck.sh"]] } +``` + +**The policy has one configuration copy, not two.** The worker never reads it: +`requestPlanVerification` calls `PlanPhaseCommands` on the coordinator and runs +only what comes back. A `verification` field in +`/etc/orchestra/worker-projects.json` would be dead configuration that later +reads as authoritative. + +### What is proven + +| Rung | Evidence | +|---|---| +| Research findings are citable | Run 10: seven findings, three confidence classes, five cited by the plan | +| The plan seals as a specification | Run 8: 7940 bytes, three phases, five `run:` lines, one manual step | +| The plan renders verbatim into implement | Run 8: the whole 7940-byte document byte for byte inside a 20822-byte `launch.md`, phase 3 heading and command present | +| Orchestra establishes phase progress | Run 8: `phase-1` verified at exit `[0]`, bound to `plan_ref`, `at_sha`, `evidence_ref`, harness and lease epoch | +| A manual step holds the phase | Runs 8, 9, 10: `awaiting_manual_verification` on every phase declaring one | +| A generic comment does not satisfy it | Run 8: "looks good to me" landed as `operator_instruction`, phase 2 unmoved | +| A keyed sign-off does | Run 8: `plan_phase_verification:51afbfa9…:phase-2`, phase 2 verified | +| Rotation preserves progress | Run 8: the successor received the whole plan and the progress block | +| Verification is bound to a SHA | Run 8: three phases rendered "verified at 14654d6cab63, stale because the tree is now at 56d9b9acdd59" | +| Reverification moves with the tree | Run 10: all three phases reverified at `ec0502f` after HEAD moved | +| A plan command outside policy is refused at seal | Run 10: refused on the planner, corrected in 25 seconds, then sealed | +| A legacy plan degrades honestly | Decoded from the real CAS artifact `1e12fcf0`: zero phases, 1646 bytes of markdown, the legacy notice present, the progress block empty | +| Review, resubmission and merge still work | Run 8: PR 11, changes requested, resubmitted, merged, receipt binds `ec320c82` | + +### What is not proven + +**The typed mismatch and the reopen.** `.orchestra/plan-mismatch.json` was never +written, across two deliberate attempts. + +Both attempts tested the wrong thing, and the agent was right both times: + +- Rewriting a file the plan depends on is *reversible*. The implementer + restored it from master and commented "Restore the byte-pinned USAGE + test_healthcheck.sh from master". The plan said the file gets no edit, an + edit appeared, and restoring it satisfied the plan. +- A human correction is *authoritative*. The brief tells the agent a decision + outranks the plan, so it followed the correction and recorded the stale plan + as an outstanding item: "The accepted plan text still specifies exit 2 and an + in-loop chain; it is stale against the correction." + +`plan-mismatch.json` is for the case where the **repository** contradicts the +plan and no human has spoken. Proving it needs a purpose-built task where the +plan asserts something false about the repo that the implementer cannot make +true. The clean route is `POST /v1/tasks//phase` from the TUI surface, +which seals an authored artifact with no intervention race: seal a plan whose +phase says to extend a helper that does not exist. + +### Defects found, all fixed + +| ID | Commit | Defect | +|---|---|---| +| F47 | `4b32080` | Delivery merges the task branch, which carries `orchestra: TASK.md`. `writeTaskFile` returned early on `os.Stat`, so every later worktree inherited the previous task's file and every release failed "TASK.md changed". Run 7 died on `retry_limit` without leaving research. | +| F48 | `49409c9` | The manual plan-phase gate had two live consumers and no producer. Every imported comment was hardcoded to `operator_instruction`, so a phase with a manual step could never be verified. A comment whose first line is `orchestra verify ` now carries the keyed subject. | +| F49 | `f4dbcf7` | A plan whose every phase was stale printed "Every phase is verified" under three lines that each said stale. | +| F50 | `4712c7d` | The research brief never stated the finding-id format the decoder enforces. Cost one boundary. | +| F51 | `98f1b2d`, `fda78cf` | The handoff parser split on `→` only and demanded the words "tried" and "failed because". Cost four leases across run 9 on content that was exactly right. | +| F52 | `fb7135e` | A plan command outside project policy sealed anyway. The only caller of `VerificationPolicy.Allows` was `PlanPhaseCommands`, which runs when the implementer asks to verify: one phase and one rotation too late. | +| F53 | `d7e75e9` | Only a verified phase consulted `Stale`. A pending manual gate rendered "automated checks passed at 94bd45c3b5d6" against a tree at `7d04aef`. | +| F54 | `015764e` | The handoff prompt stated the 200-character limit for `NEXT`, `WHY` and `REMAINING`. The validator applies it to `OPEN Q` and `LEARNED` too. | +| F55 | `38aa073` | **A refused handoff had no feedback loop.** `PrepareRelease` read the report, the parser refused it, the worker recorded it in health, and the next boundary read the same bytes. Run 10 spent four leases there. `answerRefusedHandoff` now says why and drops the file, gated on a typed `ErrInvalidHandoffAnswer` so transport failures keep their retry. | + +F50, F51 and F54 are one shape: **a constraint the code enforces and no brief +states**. F55 is the shape that made them expensive, and it is the silent-loop +pattern `CLAUDE.md` names. + +`fb7135e` has operational reach. Eight projects in `config.jsonc` carry no +`verification` block: `inference-router`, `Muzick`, `Kdrive`, `Nexus`, +`Praxis`, `Hexis`, `manga-pipeline`, `orchestra`. Their plans still seal, but +only with manual verification. A `run:` line is now refused on the planner. + +### Recorded, not fixed + +- **The operator lifecycle actions lose a version race.** `block`, `release` + and `attention` return `task version conflict` on a leased task, every + attempt, because the handler reads the version and the worker writes between + the read and the append. They succeed once the task is quiet. +- **`master` carries a stale `TASK.md`.** F47 makes it harmless, but every + pull request now shows a cosmetic `TASK.md` diff. +- **F18 is still open**, and it cost real diagnosis time here. The single + `last_error` slot held only the most recent failure, so the three earlier + ones in run 7 were overwritten before they could be read. + +### Things that will bite + +- The agent pane is on a private tmux socket: + `tmux -L orchestra capture-pane -p -t :1.0`. A bare `tmux ls` does + not show it. +- A trivial task finishes three plan phases in about 105 seconds. Any + intervention timed against the implement phase will lose that race. +- `sudo` is unavailable in this sandbox, so every worker-side fix needs the + operator. Batch them: one install per run, not one per defect.