kami
b317ecb1bd
F57 accepted, F58 and F59 abandoned. The rig, the two live traces, and the correction to the operator-lifecycle entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2277 lines
100 KiB
Markdown
2277 lines
100 KiB
Markdown
# Burn-in: live conformance, not more architecture
|
||
|
||
Written 2026-08-26. The workflow is feature-complete enough to exercise. What
|
||
remains is empirical: run the same real task shape through each harness and
|
||
classify what breaks. Isolated tests are insufficient by design here, and only
|
||
the live owner path establishes conformance.
|
||
|
||
Do not add workflow features while this is running. Findings decide the next
|
||
implementation work.
|
||
|
||
## Burn-in build identity
|
||
|
||
`77a2b323fabcf080d7542061ae2d7b3c34eef5b7`
|
||
|
||
Both halves must report exactly this revision before a task is created. Neither
|
||
needs a credential now: the coordinator prints it in `docker logs orchestra-api`
|
||
and the worker in `journalctl -u orchestra-worker`. The same object is at
|
||
`GET /v1/admin/diagnostics` and `GET /v1/federation/workers` behind the operator
|
||
login.
|
||
|
||
Build both with `deploy/build.sh <outdir>`, which refuses a dirty tree. Later
|
||
documentation-only commits do not change this identity, so a rebuild either
|
||
passes this revision explicitly or accepts the new one and redeploys both
|
||
halves. Never one half.
|
||
|
||
## Deployment state, 2026-08-26 18:35
|
||
|
||
| Half | State |
|
||
|---|---|
|
||
| Coordinator (homesrv) | **Deployed at 6f9300b.** Rebuilt with `--build-arg BUILD_REVISION`, recreated, `/readyz` ready, self-reporting the revision in its log |
|
||
| Worker (workpc) | **Staged, not installed.** `~/orchestra-deploy/orchestra-worker` sha256 `2b1c43071eaf6b5c15b110de39f204038a9629897e0ce3dacf45be96a6e6529e`. Installed binary is still the 2026-07-30 build |
|
||
|
||
Two operator steps remain, both needing root:
|
||
|
||
```sh
|
||
sudo install -m 0755 /home/kami/orchestra-deploy/orchestra-worker /usr/local/bin/orchestra-worker
|
||
sudo systemctl restart orchestra-worker
|
||
journalctl -u orchestra-worker -n 5 --no-pager # must print revision 6f9300b...
|
||
```
|
||
|
||
A third blocker, found while preparing the pane check:
|
||
|
||
**herdr is not running on workpc.** `herdr status server` reports `not running`,
|
||
the socket at `/home/kami/.config/herdr/herdr.sock` refuses connections, and its
|
||
log stops at 2026-07-30. `workpc-opencode` therefore cannot start a pane, even
|
||
with the worker installed. The worker logs `serving harness workpc-opencode
|
||
(opencode) on herdr backend` at startup without touching the socket, so that
|
||
line is not evidence the backend is reachable. herdr is an interactive terminal
|
||
workspace manager: running `herdr` in a terminal launches or attaches to the
|
||
persistent session and starts the server. Confirm with `herdr status server`
|
||
before creating a task.
|
||
|
||
Then scrub the pane environment before letting an agent run.
|
||
|
||
### Where the pane environment actually comes from
|
||
|
||
The two workpc harnesses inherit different environments, so one scrub does not
|
||
cover both.
|
||
|
||
- **`workpc-opencode` (herdr backend).** The pane is created by the herdr
|
||
daemon, which is a separate long-running process. It inherits herdr's
|
||
environment, not the worker's. Scrubbing `worker.env` does nothing here.
|
||
Whatever environment herdr is started with is what every opencode pane gets.
|
||
- **`workpc-claude` (tmux backend).** `TmuxBackend.StartAgent` runs `tmux
|
||
new-session` through `exec.CommandContext` with no `Env` set
|
||
(`internal/herdr/tmux.go`). If the tmux server is not already up, the worker
|
||
starts it, and that server inherits the worker's full environment. Every
|
||
claude pane then inherits it too.
|
||
|
||
This exposes a conflict the scrub alone cannot resolve. The worker legitimately
|
||
needs `ORCHESTRA_WORKER_TOKEN` (or `ORCHESTRA_WORKER_TOKEN_<ID>`) and
|
||
`ORCHESTRA_FEDERATION_ADMIT_TOKEN`, and deleting them breaks the worker. Keeping
|
||
them means the tmux path hands them to the agent. Closing it needs either a
|
||
filtered `cmd.Env` when the backend spawns a pane, or a tmux server started
|
||
separately with a clean environment. Not on flow 1's path, because flow 1 is
|
||
opencode only. Close it before step 6.
|
||
|
||
## Baseline: one harness, one path
|
||
|
||
`workpc-opencode` only. Codex is a coverage gap, not a prerequisite, and no
|
||
worker runs on homesrv. Prove one real path first:
|
||
|
||
```
|
||
real vikunja task
|
||
→ workpc-opencode
|
||
→ frame/research/plan
|
||
→ trajectory gate if configured
|
||
→ implement
|
||
→ independent review
|
||
→ task pr
|
||
→ human merge
|
||
→ completed
|
||
```
|
||
|
||
Then raise difficulty in this order:
|
||
|
||
```
|
||
1. opencode boring success
|
||
2. opencode mid-session human correction
|
||
3. opencode forced rotation
|
||
4. opencode pr rejection -> fix -> merge
|
||
5. opencode reconcile outage/recovery
|
||
6. representative cases on claude
|
||
7. add the codex adapter/config
|
||
8. cross-harness rotation: opencode -> claude/codex
|
||
```
|
||
|
||
## Herdr transports, so nobody misdiagnoses again
|
||
|
||
No entry in the live `config.jsonc` sets `backend` or `address`, so all six
|
||
resolve to `<machine>:9245` (`registry.defaultHerdrPort`). Both ports are
|
||
closed. That says nothing about the workpc harnesses: they are worker-owned,
|
||
and the coordinator logs `worker-owned on workpc; coordinator probe skipped`
|
||
for each. The worker reaches `workpc-claude` over a tmux socket and
|
||
`workpc-opencode` over `/home/kami/.config/herdr/herdr.sock`. Do not diagnose
|
||
harness availability from a TCP probe of 9245.
|
||
|
||
## Evidence to record per run
|
||
|
||
One row per run. The launch instruction is now dumped to
|
||
`<worktree>/.orchestra/launch.md` (`herdr.LaunchContextFile`) at every launch,
|
||
local and federated, so the context is auditable without reading pane
|
||
scrollback.
|
||
|
||
```
|
||
harness (claude | codex | opencode)
|
||
task id
|
||
lease epoch (every epoch, if the task rotates)
|
||
session ids (pane ids per epoch)
|
||
human decision ids
|
||
launch context (sha256 of .orchestra/launch.md, per launch)
|
||
handoff ref
|
||
git sha (at every boundary: launch, each handoff, review, submit, merge)
|
||
review ref
|
||
submission ref
|
||
pr id
|
||
completion receipt
|
||
```
|
||
|
||
## The five flows
|
||
|
||
1. **Boring success.** `task → research → plan → implement → review → pr → merge`.
|
||
2. **Mid-session correction.** Agent is doing A, the human says B, the next
|
||
verified boundary delivers B, and A becomes history rather than a competing
|
||
instruction.
|
||
3. **Rotation.** Agent A hands off, agent B resumes. Switch harnesses across the
|
||
boundary if both are up.
|
||
4. **Human rejection.** Submit sha A, comment on the pull request, task reopens,
|
||
fix at sha B, a fresh review, the same pull request, merge.
|
||
5. **Failure path.** Human source goes down, the reconcile streak escalates, the
|
||
session hands off, the successor lease is refused, the source recovers, and a
|
||
corrected successor starts.
|
||
|
||
## What to inspect by hand
|
||
|
||
For every launch, read `.orchestra/launch.md` and ask only:
|
||
|
||
```
|
||
does this agent know
|
||
- what the task actually wants?
|
||
- what was most recently decided?
|
||
- what phase it is in?
|
||
- what earlier material is merely historical?
|
||
- what to do next?
|
||
```
|
||
|
||
Then compare that against what the model did.
|
||
|
||
## Classify before fixing
|
||
|
||
Every failure gets exactly one label before any code is written:
|
||
|
||
```
|
||
authority bug the wrong thing outranked the right thing
|
||
context-selection bug the renderer showed or hid the wrong material
|
||
lifecycle bug state, lease, phase or event handling is wrong
|
||
adapter/harness bug the pane, occupancy, boundary or prompt path is wrong
|
||
model-following failure the context was right and the model ignored it
|
||
operator-policy gap credentials, deployment or configuration
|
||
```
|
||
|
||
This matters because the cheap response to every failure is another prompt
|
||
rule, and prompt rules accumulated to compensate for lifecycle or adapter bugs
|
||
are how the enforcement boundary rots. A model-following failure is the only
|
||
class a prompt change should ever answer.
|
||
|
||
## Pane credential cleanup (operator, in parallel)
|
||
|
||
The agent pane inherits the worker's environment. On workpc the worker is
|
||
`orchestra-worker.service`, `User=kami`, `EnvironmentFile=/etc/orchestra/worker.env`
|
||
(mode 0600, root-owned, deliberately unreadable here). Remove from that file, and
|
||
from anything else the pane inherits:
|
||
|
||
```
|
||
gitea mutation token
|
||
vikunja mutation credentials
|
||
orchestra operator / system token
|
||
```
|
||
|
||
Retain only `ORCHESTRA_AGENT_TOKEN` and credentials a specific task genuinely
|
||
needs. If an agent needs a forge operation, it asks Orchestra to perform it.
|
||
|
||
Then verify from inside a real pane:
|
||
|
||
```bash
|
||
git push # must fail unless Orchestra supplied the auth
|
||
curl .../v1/tasks/<id>/submission # must be 403 for the agent surface
|
||
tea pr create # must have no usable mutation credential
|
||
```
|
||
|
||
Note what is *not* enforced by code: nothing in this repo scrubs the pane
|
||
environment, and a fake `git` earlier in `$PATH` is not enforcement because
|
||
`/usr/bin/git` bypasses it. Credential isolation is the enforcement.
|
||
Confinement of network, filesystem and destructive commands belongs to whatever
|
||
launches the process (herdr, a container, systemd, bubblewrap), not to
|
||
Orchestra, which supplies identity and policy inputs.
|
||
|
||
## Exit criterion
|
||
|
||
Roughly 10 to 20 real tasks, with the five flows covered on each harness that
|
||
is actually up. Then read the classified failures and decide the next
|
||
implementation unit from them.
|
||
|
||
## Live findings, 2026-08-26 21:35
|
||
|
||
Both halves are paired at `6f9300b`, and herdr is up (0.7.5, protocol 17). What
|
||
remains before flow 1, and what the in-pane credential checks will really show.
|
||
|
||
### Flow 1 has no repository to run in
|
||
|
||
`workpc-opencode` declares exactly one project, `test-e2e`, pointing at
|
||
`/tmp/test-e2e` and `/tmp/test-e2e-worktrees`. Neither path exists on workpc
|
||
after today's reboot, and `/var/lib/orchestra/repos` does not exist there at
|
||
all. Nothing can be leased to this harness until it has a project backed by a
|
||
durable local repository.
|
||
|
||
The only configured task source is Gitea `kami/correx` at
|
||
`https://gitea.kvmx.ru` (`ORCHESTRA_GITEA_URL/OWNER/REPO`), so a real task means
|
||
a `correx` issue. `correx` already has `machine_affinity: ["homesrv","workpc"]`
|
||
in the coordinator's `config.jsonc`. What is missing is a workpc-local entry:
|
||
|
||
```jsonc
|
||
// /etc/orchestra/worker-projects.json, on workpc
|
||
{
|
||
"correx": {
|
||
"repo": "/home/kami/orchestra/repos/correx.git",
|
||
"worktree_root": "/home/kami/orchestra/worktrees/correx",
|
||
"remote": "origin"
|
||
}
|
||
}
|
||
```
|
||
|
||
Not `/tmp`. The burn-in outlives a reboot.
|
||
|
||
### There is no vikunja ingest
|
||
|
||
This repo has no Vikunja provider. `/readyz` reports `gitea` and `jsonl`. A
|
||
"real vikunja task" cannot enter Orchestra today: it arrives as a Gitea issue or
|
||
through the JSONL watcher.
|
||
|
||
### `git push` from a pane will succeed, and no Orchestra setting stops it
|
||
|
||
Corrected 21:45. The mechanism is SSH, not HTTPS. HTTPS has no stored
|
||
credential on workpc, where `git push` over `https://` fails with `could not
|
||
read Username`. Pushing works over `ssh://git@gitea.kvmx.ru:2222` using kami's
|
||
default RSA identity, which Gitea lists as the key named `workpc`. It carries no
|
||
passphrase, so no agent is needed.
|
||
|
||
That identity is what lets the *worker* push, and panes run as the same user
|
||
with the same home directory, so an agent inherits it. `worker.env` is
|
||
irrelevant to this. Real isolation needs panes under a different unix user.
|
||
Expect `git push --dry-run` to succeed from a pane, and record it as an
|
||
operator-policy gap rather than a code defect.
|
||
|
||
The worker inherits ambient Git credentials by design: `git()` in
|
||
`cmd/orchestra-worker/main.go` runs `exec.CommandContext` with no environment of
|
||
its own. There is no separate credential path for the worker to hold something
|
||
the pane does not.
|
||
|
||
### The agent surface is currently unauthenticated
|
||
|
||
`ORCHESTRA_AGENT_TOKEN` is unset on the coordinator, and an unset surface token
|
||
means the middleware performs no check for that surface. Probed live:
|
||
|
||
```
|
||
POST /v1/tasks/<id>/decision-request -H 'X-Orchestra-Surface: agent' -> 404 (reached the handler)
|
||
POST /v1/tasks/<id>/phase -H 'X-Orchestra-Surface: agent' -> 403 (refused)
|
||
```
|
||
|
||
The capability boundary holds. Authentication does not. Set
|
||
`ORCHESTRA_AGENT_TOKEN` in the coordinator `.env` before the burn-in, and note
|
||
the same is true of `ORCHESTRA_MCP_TOKEN` and `ORCHESTRA_MAVEN_TOKEN`, both
|
||
unset.
|
||
|
||
### Two more unset settings that change burn-in behaviour
|
||
|
||
- `ORCHESTRA_FEDERATION_ADMIT_TOKEN` is unset, so any caller may register a new
|
||
worker identity. Existing identities stay protected, because registering an
|
||
existing id with a different token is refused.
|
||
- `ORCHESTRA_REVIEW_ACTORS` is unset, which `human.Trust` reads as "anyone not
|
||
explicitly ignored". Flow 4 will accept a task-moving comment from any Gitea
|
||
actor. Set it to `kami`.
|
||
|
||
## Flow 1 setup, 2026-08-26 21:45
|
||
|
||
Target: `kami/test-e2e` on Gitea, a throwaway repo, rather than `correx`.
|
||
|
||
Done:
|
||
|
||
- **Ingest switched.** `ORCHESTRA_GITEA_REPO=test-e2e` in the coordinator `.env`
|
||
(previous file kept as `.env.pre-burnin-20260826`), container recreated, still
|
||
reporting `6f9300b`. The project id must equal the repo name, because
|
||
`main.go` sets `Project: os.Getenv("ORCHESTRA_GITEA_REPO")`. `test-e2e`
|
||
already exists in `config.jsonc` with `machine_affinity: ["workpc"]`.
|
||
- **Durable repo on workpc.** Bare clone at
|
||
`/home/kami/orchestra/repos/test-e2e.git`, worktree root
|
||
`/home/kami/orchestra/worktrees/test-e2e`. Origin points at
|
||
`ssh://git@gitea.kvmx.ru:2222/kami/test-e2e.git`, and `git push --dry-run`
|
||
reports `Everything up-to-date`. The old `/tmp` paths did not survive the
|
||
reboot and must not come back.
|
||
|
||
Remaining, needs root:
|
||
|
||
```jsonc
|
||
// /etc/orchestra/worker-projects.json
|
||
{
|
||
"test-e2e": {
|
||
"repo": "/home/kami/orchestra/repos/test-e2e.git",
|
||
"worktree_root": "/home/kami/orchestra/worktrees/test-e2e",
|
||
"remote": "origin"
|
||
}
|
||
}
|
||
```
|
||
|
||
Then `sudo systemctl restart orchestra-worker`.
|
||
|
||
Store state, checked before the first run: 30 tasks, all `test-e2e`, with 23
|
||
blocked, 6 completed and 1 failed. All are July leftovers and none holds a
|
||
lease. The coordinator has run `ResumeAnsweredBlockers` every second for hours
|
||
without resuming any of them, so they are inert rather than merely quiet. The
|
||
July stuck task `06FT6CKD9Y98AZRX6X8K3QXFZG` is now `failed`.
|
||
|
||
A stale branch `orchestra/scratch/oc-06ftgkjadcd2hwjn2zwjen90q4` exists on the
|
||
remote from an earlier run. Harmless, but it is not from this burn-in.
|
||
|
||
## Run 1: task 06G3YR34117MAYT6KEAC9RJHD0, 2026-08-26
|
||
|
||
Evidence ledger:
|
||
|
||
```
|
||
harness workpc-opencode (herdr backend)
|
||
task 06G3YR34117MAYT6KEAC9RJHD0
|
||
source gitea:test-e2e/1 "Add a --version flag to the healthcheck script"
|
||
attempts 3 leases: two nacked, third launched
|
||
pane wN:p1, agent oc-06g3yr34117mayt6keac9rjhd0
|
||
agent session ses_fc096de31ffelZeMngQEyBTxfh
|
||
worktree /tmp/test-e2e-worktrees/06G3YR34117MAYT6KEAC9RJHD0
|
||
branch orchestra/06G3YR34117MAYT6KEAC9RJHD0
|
||
head at launch 5e6c4783d4213a934dc486160b888b6014bc2d03
|
||
launch context .orchestra/launch.md, 2138 bytes
|
||
phase frame
|
||
decisions none
|
||
```
|
||
|
||
### What worked
|
||
|
||
- **The launch context is right.** Authority order, the ambiguity ladder, the
|
||
phase brief with `frame: ... Do not change code`, `Current human decisions:
|
||
None recorded`, and a verified git state with worktree, branch and head.
|
||
- **The agent respected the phase.** `agent_status: done`, no code changes, only
|
||
Orchestra's own scratch directory in the tree.
|
||
- **A bad launch nacked cleanly.** The first attempt failed on the base checkout
|
||
and the task went back to `queued` with `lifecycle_phase: launch_nacked` and
|
||
`attempt: 1`. No orphan pane, no stuck lease.
|
||
|
||
### Fixed during the run
|
||
|
||
- **F1, authority bug, `2753a8d`.** Every federated launch died with `effective
|
||
intent: federation: 401 Unauthorized: unauthorized surface`.
|
||
`GET /v1/tasks/<id>/intent` exists for workers, and the authz worker-path
|
||
exemption never included it. Twenty test packages passed throughout.
|
||
- **F2, authority bug, `09e572f`.** The rendered goal was the issue title alone
|
||
and acceptance read `Not stated.`, because `Gitea.event` parsed the issue body
|
||
and dropped it from the `TaskCreated` payload. The store already read
|
||
`description`. Every Gitea-sourced task so far ran on its title.
|
||
- **F3, self-inflicted, `a0209a2`.** The launch dump left `.orchestra/`
|
||
untracked in a worktree whose own context said `uncommitted changes: false`.
|
||
It would have polluted the gate, the review diff and the agent's `git status`.
|
||
|
||
### Open, classified
|
||
|
||
- **F4, adapter/harness.** `rotation ... activity degraded: activity unknown:
|
||
adapter: resolve session file: adapter: harness "opencode" has no
|
||
session-file resolver`. Thrash and activity triggers are permanently degraded
|
||
on opencode. Observable in the worker's `last_error`, non-fatal.
|
||
- **F5, lifecycle.** The router never leased this task. Every gate checks out by
|
||
hand: worker `online: true`, `herdr_status: reachable`, fresh `checked_at`,
|
||
`supported_projects: ["test-e2e"]`, quota empty, no active leases, capability
|
||
empty. A direct `POST /v1/tasks/<id>/lease` succeeded instantly. The refusal
|
||
is upstream of `Store.Lease` and silent by construction
|
||
(`internal/router/router.go:197`). All three runs were leased by hand.
|
||
- **F6, observability.** Worker health reports `active_task: null` while the
|
||
store shows the task leased and herdr shows a live opencode agent in the
|
||
task's worktree.
|
||
- **F7, operator policy.** `ORCHESTRA_TUI_TOKEN` is unset, and an unset surface
|
||
token means no authentication for that surface. That is how the manual leases
|
||
above were issued, unauthenticated, from another machine. The TUI surface is
|
||
`FullControl`, so this is the whole control plane, not just the two agent
|
||
request endpoints. Set the token.
|
||
- **F8, correctness.** `human.Reconciler.Reconcile` iterates every configured
|
||
source for every task, so the three queued `correx` tasks have their external
|
||
ids looked up in `kami/test-e2e`. A source should only reconcile the tasks
|
||
that came from it. Likely why those three never lease.
|
||
|
||
## Fix pass before run 2, 2026-08-26 23:50
|
||
|
||
Burn-in identity is now `77a2b32`. The coordinator is deployed at it. The worker
|
||
is staged at it, sha256 `2a850f2d1102d390...`, still needing root to install.
|
||
|
||
### Closed
|
||
|
||
- **F7, security.** `authz.RequireCredentials` refuses startup when a
|
||
full-control surface has no token, rather than logging it.
|
||
`ORCHESTRA_TUI_TOKEN` is set in the coordinator `.env`. Verified live: an
|
||
unauthenticated `POST` on the TUI surface now returns 401. Web is exempt
|
||
because `Sessions` makes its login mandatory. `ORCHESTRA_MCP_TOKEN`,
|
||
`ORCHESTRA_MAVEN_TOKEN` and `ORCHESTRA_AGENT_TOKEN` remain unset, so those
|
||
surfaces are still unauthenticated for reads and for their three request
|
||
endpoints. Bounded by capability, worth closing, not startup-fatal.
|
||
- **F5, lifecycle.** Every eligibility gate now records a `router.Rejection`,
|
||
exposed at `GET /v1/router/health` and reset per pass. No gate was weakened.
|
||
The live output immediately explained the three stuck `correx` tasks:
|
||
`worker has not declared project correx`. A queued task in retry backoff was
|
||
skipped before the candidate loop and recorded nothing at all, which is the
|
||
shape that hid the original case; it now reports
|
||
`retry backoff until <time>`.
|
||
- **F8, correctness.** Reconciliation is bound to `task.Source`, the
|
||
`provider:project` identity the ingest stamped. A source that cannot prove it
|
||
owns the task is skipped, and a task with no matching source reconciles to
|
||
nothing and still launches. The integration fixture had encoded the bug: it
|
||
ingested from `jsonl` and reconciled from `gitea`.
|
||
- **F11, outage.** The API entered a restart loop exiting with `invalid event:
|
||
until_ns required`. `ValidateEvent` compared `until_ns` against `time.Now()`
|
||
for `TaskLeased` and `TaskLeaseRenewed`, so a lease event that was valid when
|
||
written failed validation once it expired. `store.Open` replays the tail after
|
||
the snapshot and `log.Fatal`s on the first invalid event, so the coordinator
|
||
refused its own history. Validation of a durable event is now
|
||
time-independent. Latent since the field existed: it needed a renewal in the
|
||
post-snapshot tail plus a restart after that renewal expired.
|
||
|
||
### Also found
|
||
|
||
- **F9, lifecycle.** An operator cannot release a leased task through
|
||
`POST /v1/tasks/<id>/release` without knowing its `harness_id` and
|
||
`lease_epoch`, because `Store.Append` fences lifecycle events on a leased
|
||
task and the endpoint passes the request body through unchanged. Correct that
|
||
only the owner releases, but there is no operator escape hatch.
|
||
- **F10, hygiene.** A coordinator-side release under a live worker leaves the
|
||
worker renewing a lease it no longer holds. Combined with F11 that produced a
|
||
per-second invalid-event log line.
|
||
|
||
### Shared checkout
|
||
|
||
Another session is working on the auth and frontend layers in this same
|
||
checkout. Two consequences, both handled: `deploy/build.sh` and the container
|
||
image now build in a detached worktree of the revision they stamp, so no
|
||
uncommitted work is compiled into a stamped binary and no in-progress edit
|
||
blocks a deploy. Commits from this session name their paths rather than using
|
||
`git add -A`. The first commit, `7f12c7f`, predates that discipline and swept in
|
||
whatever was uncommitted at the time, including `web/src` and `internal/authn`.
|
||
|
||
### Run 1 is closed as a diagnostic
|
||
|
||
Task `06G3YR34117MAYT6KEAC9RJHD0` was released with reason `abandoned diagnostic
|
||
run`, supplying the lease fence by hand per F9. It sits at `attempt: 3` and the
|
||
router will fail it on its next pass. It is not a conformance run: it needed
|
||
three manual leases and its authority was built before the issue-body fix.
|
||
|
||
### Run 2 preconditions
|
||
|
||
```
|
||
worker installed at 77a2b32 and restarted
|
||
coordinator at 77a2b32 done
|
||
F7 closed done
|
||
F5 exposed done
|
||
F8 fixed done
|
||
fresh issue, no manual leasing
|
||
```
|
||
|
||
## Runs 2 and 3, 2026-08-27
|
||
|
||
Written from live evidence. Neither run was a conformance pass, and both were
|
||
worth more than one: run 2 found three blocking bugs behind each other, and
|
||
run 3 found two more plus the reason six isolated probes disagreed with
|
||
production.
|
||
|
||
`AUDIT.md` is uncommitted and owned by another session, so the burn-in ledger
|
||
lives here.
|
||
|
||
### Fixed, with the revision each landed in
|
||
|
||
- **F12**, lifecycle, `0d67af9`. `Store.QuotaSince` reported an empty window as
|
||
unknown, `QuotaAvailability` fails closed on unknown, and every herdr in
|
||
`config.jsonc` declares a quota limit. The only producer of a receipt is a
|
||
completed lease, so nothing could ever be leased. An empty window is now
|
||
observable zero. A receipt that declares its own consumption unknown still
|
||
fails closed. Verified live: run 2 leased three seconds after the fixed
|
||
coordinator started.
|
||
- **F13**, observability, `0d67af9`. `federatedAvailability` restated a quota
|
||
refusal as worker health, so router health said `stale heartbeat` against a
|
||
heartbeat one second old. Gates name themselves through
|
||
`router.ReasonedAvailability`. Verified live.
|
||
- **F14**, authority, `a5d361b`. Gitea ingest never set `acceptance`. One
|
||
recognized heading, bullets and checkboxes until the next heading, order
|
||
preserved, section removed from the description. Verified live on run 3: five
|
||
ordered items, prose above the heading kept as the description.
|
||
- **F15**, adapter, `1fd82f8`. Two bugs. `TmuxBackend.Prompt` wrote the whole
|
||
instruction with `send-keys -l`, Claude Code coalesced it into a paste, and
|
||
the Enter was absorbed. And `TaskLaunchAcknowledged` meant "Prompt returned
|
||
nil", not "the harness accepted it". Launch transport became a backend
|
||
property, and `ConfirmLaunch` began polling for proof. The detection works.
|
||
The fix did not: see F17.
|
||
- **F16**, lifecycle, `1888d42`. `renewLeases` renewed whenever a session
|
||
existed and `PaneCapture` succeeded, so a pane that opened and never started
|
||
renewed forever. This is why the July task stayed orphaned: F15 explains why
|
||
nothing started, F16 why the lease never let go. Renewal now needs the agent
|
||
busy, or the pane capture to differ from the hash recorded at the previous
|
||
renewal. `lease.ProgressSHA` carries that hash.
|
||
- **F17**, adapter, `f54fb00`. `pendingInput` scanned every line beginning with
|
||
the prompt marker, but queued and already-accepted input renders with the
|
||
same prefix. Only the editor owning the pane cursor is unsubmitted, so
|
||
`inputState` reads `#{cursor_y}` and captures screen rows without `-J`, which
|
||
would invalidate the row index. On that footing `ConfirmLaunch` became an
|
||
active submit protocol: resend Enter while the live editor still holds
|
||
exactly what was submitted, at most three times, no closer than two poll
|
||
intervals, then observe until the deadline. Queued input confirms rather than
|
||
fails. Evidence records `confirmation`, `submit_attempts` and both timestamps.
|
||
- **F19**, lifecycle, `edff021`. A task that reached the router's
|
||
`MaxAttempts` was permanently terminal. `TaskReleased` only increments
|
||
`Attempt`, `TaskCorrected` could not touch it, and no HTTP route emitted a
|
||
correction. `POST /v1/tasks/{id}/retry` requires the task to be failed,
|
||
unleased, and failed with `reason: retry_limit`, then appends one
|
||
`TaskCorrected` naming that failure with `state: queued` and `attempt: 0`.
|
||
Identity, goal, acceptance, decisions, phase and artifact refs all survive,
|
||
and the original failures stay in the log. `operation_id` is required and
|
||
makes it idempotent. Deliberately not a generic correction endpoint.
|
||
|
||
### The finding that mattered most
|
||
|
||
Run 3 failed three times with `prompt_not_submitted`, and six isolated probes
|
||
of the same code path could not reproduce it: fresh session, untrusted
|
||
directory with the trust dialog, `launch.md` present before startup, 100ms
|
||
readiness polling, `env -i` with only the worker's systemd variables, and the
|
||
real `StartAgent`/`Prompt`/`ConfirmLaunch` against a byte-identical git
|
||
worktree. All six submitted on the first Enter.
|
||
|
||
F17 turned that disagreement into data instead of a theory. The first live
|
||
launch after it deployed:
|
||
|
||
```
|
||
launch 06G44JZB80MZBEY97196EZN8EC confirmed: confirmation=editor_cleared
|
||
submit_attempts=2 first_submit_at=2026-08-27T09:12:00.801855937Z
|
||
confirmed_at=2026-08-27T09:12:01.315587139Z
|
||
```
|
||
|
||
Production needs the second Enter. Isolated probes need one. The submit is not
|
||
deterministic, which is exactly what F15's `ConfirmLaunch` comment asserted it
|
||
was.
|
||
|
||
### Still open
|
||
|
||
- **F18**, observability. A queued launch used to be misread as unsubmitted.
|
||
F17 fixes the predicate, but nothing else that parses pane text distinguishes
|
||
the active editor from history, so the same class of error can recur wherever
|
||
`PaneCapture` output is matched.
|
||
- **F9** is unchanged by F19. An operator still cannot release a lease someone
|
||
else owns without supplying that owner's `harness_id` and `lease_epoch`,
|
||
which means reading them out of the event log first. F19 recovers a terminal
|
||
task, which is a different escape hatch.
|
||
- **F6**, observability. Worker health reported `active_task: null` while the
|
||
store showed the task leased. Run 2 showed this was partly honest, because
|
||
the worker really had no working agent. Recheck now that launches confirm.
|
||
- **F4**, adapter quality. `harness "opencode" has no session-file resolver`,
|
||
so activity and thrash triggers are degraded on opencode.
|
||
- **F10**, hygiene. A coordinator-side release under a live worker leaves the
|
||
worker renewing a lease it no longer holds.
|
||
- Unset gated tokens: `ORCHESTRA_MCP_TOKEN`, `ORCHESTRA_MAVEN_TOKEN`,
|
||
`ORCHESTRA_AGENT_TOKEN`.
|
||
- `gofmt -l` fails on `internal/provider/provider.go`,
|
||
`internal/router/router.go` and `internal/webui/webui.go` at `edff021`. None
|
||
were touched by these runs. `internal/router/router.go` picked it up in
|
||
`0d67af9`. `go vet` passes, which is why nobody noticed.
|
||
|
||
### Operator actions taken by hand, and why
|
||
|
||
- Task `06G3ZCZWJ3QHF992ZMDGSJ0PYG`, run 2, was blocked with
|
||
`block_reason: operator_block` rather than released. A released task returns
|
||
to `queued`, and the router leases `queued` tasks, so releasing it would have
|
||
put it in competition with run 3.
|
||
- The first block attempt returned `task version conflict`.
|
||
`Store.validateTransition` fences every lifecycle event on a leased task, so
|
||
the payload needs `harness_id` and `lease_epoch` even though the HTTP handler
|
||
does not ask for them. That is F9 in practice.
|
||
- Task `06G44JZB80MZBEY97196EZN8EC`, run 3, was recovered with F19's retry
|
||
rather than by filing a fourth issue. That keeps `kami/test-e2e#3` served by
|
||
its own task and proves terminal recovery in the same run.
|
||
|
||
### Deployment boundary
|
||
|
||
Both halves report `edff021`, built from a detached worktree of that revision.
|
||
Worker sha256 `68ac265455acb0e007bfb0e898a91b317cd7289d7f2bf5da6df1224e1d6706a5`
|
||
at `/usr/local/bin/orchestra-worker`. Coordinator image built 13:08:30 +0400.
|
||
|
||
## Run 3: autonomous expiry and relaunch, 2026-08-27 11:02 UTC
|
||
|
||
Task `06G44JZB80MZBEY97196EZN8EC`, issue `kami/test-e2e#3`. Observed live, no
|
||
operator action in the window. The stranded epoch was
|
||
`06G44ZBZX4YH6PRN7Y4ZH8GG3W`.
|
||
|
||
| seq | at (UTC) | event | epoch |
|
||
|---|---|---|---|
|
||
| 358 | 10:32:09 | `TaskLeaseRenewed` v15 | `06G44ZBZX4YH6PRN7Y4ZH8GG3W` |
|
||
| 359 | 11:02:16 | `TaskReleased` v16, `reason: lease_expired` | `06G44ZBZX4YH6PRN7Y4ZH8GG3W` |
|
||
| 360 | 11:03:17 | `TaskLeased` v17 | `06G45RVKAW20H1D22NHBRYKMN8` |
|
||
| 361 | 11:03:22 | `TaskLaunchAcknowledged` v18, `lifecycle_phase: started` | `06G45RVKAW20H1D22NHBRYKMN8` |
|
||
|
||
Expiry to acknowledged launch: 66 seconds, autonomous. Every event from seq 353
|
||
onward carries `surface: system`. The last `tui` event is the `TaskCorrected` at
|
||
09:11:55, before the window.
|
||
|
||
Launch receipt, worker journal, 15:03:22 +04:
|
||
|
||
```
|
||
launch 06G44JZB80MZBEY97196EZN8EC confirmed: confirmation=editor_cleared submit_attempts=2 first_submit_at=2026-08-27T11:03:22.203859355Z confirmed_at=2026-08-27T11:03:22.717657436Z
|
||
```
|
||
|
||
### F16 is partial, not closed
|
||
|
||
The hard predicate passed: no renewal carried the stranded epoch after
|
||
10:45:26. The run did not exercise the progress comparison. Worker health
|
||
records why the renewal was skipped:
|
||
|
||
```
|
||
"last_error": "validate lease 06G44JZB80MZBEY97196EZN8EC: tmux capture-pane -p -J -S -200 -t =orchestra-06g44jzb80mzbey97196ezn8ec-e53607ea:1.0: no server running on /tmp/tmux-1000/orchestra: exit status 1",
|
||
"error_at": "2026-08-27T11:02:16.213318366Z"
|
||
```
|
||
|
||
That is the `paneProgress` error path at `cmd/orchestra-worker/main.go:973`, not
|
||
the `default` branch at line 992. The tmux server was gone, so no pane text
|
||
could be hashed.
|
||
|
||
- **Proven:** dead or missing pane, no renewal, lease expires, autonomous
|
||
re-lease and relaunch.
|
||
- **Unproven:** live pane with an idle agent and unchanged output, renewal
|
||
refused.
|
||
|
||
### Epoch reset, stated precisely
|
||
|
||
The new epoch and its acknowledged launch establish the reset structurally.
|
||
`leases` in `/var/lib/orchestra/worker-state/workpc-claude.json` holds only
|
||
`{epoch, version, until}`, with `progress_sha` absent under `omitempty`. No
|
||
runtime value of `ProgressSHA`, `UsageBaseline` or `PickupAcknowledged` was
|
||
observed.
|
||
|
||
### F6 closed
|
||
|
||
Worker health reports `active_task_id: 06G44JZB80MZBEY97196EZN8EC` with
|
||
`active_pane_id` set, against a leased task.
|
||
|
||
### F18 sharpened, non-blocking
|
||
|
||
`recordError` (`cmd/orchestra-worker/main.go:70`) keeps one slot, so the earlier
|
||
renewal decision near 10:52 was overwritten and cannot be read. One
|
||
`last_error` slot cannot preserve a causal sequence. Later replacement: a
|
||
bounded recent-error ring or event-backed observations, roughly the last 8
|
||
`{at, class, message}` entries. Do not land this during a live run.
|
||
|
||
### Ledger after this checkpoint
|
||
|
||
```
|
||
F6 closed
|
||
F14 closed
|
||
F15 closed by detection, transport fix pending live proof
|
||
F16 partial: missing-pane branch proven, static-live-pane branch unproven
|
||
F17 fixed, isolated live proof
|
||
F18 open observability
|
||
F19 fixed
|
||
F20 awaiting first post-frame confirmed input
|
||
```
|
||
|
||
### Next decisive checkpoint, F20
|
||
|
||
```
|
||
frame completes
|
||
→ Orchestra emits the next input
|
||
→ worker journal must contain: input to <pane> confirmed: confirmation=<kind> submit_attempts=<n>
|
||
→ only then may the agent continue
|
||
```
|
||
|
||
Receipt appears and the model proceeds: let the run continue through phase
|
||
transitions. Input lands with no receipt: F20 fails. No input attempted after
|
||
`frame`: a different lifecycle or phase-advance bug.
|
||
|
||
## Run 3 conformance result, 2026-08-27 12:00 UTC
|
||
|
||
```text
|
||
run 3 conformance result: failed
|
||
cause: no autonomous phase-advance path
|
||
```
|
||
|
||
### F16 closed
|
||
|
||
```text
|
||
missing-pane branch: proven
|
||
live-pane idle + unchanged output branch: proven
|
||
external editor input ignored as progress: proven
|
||
```
|
||
|
||
Second branch evidence, worker health at 11:53:46 with the pane alive and
|
||
`herdr_status: reachable`:
|
||
|
||
```
|
||
"last_error": "lease 06G44JZB80MZBEY97196EZN8EC not renewed: agent status idle and pane unchanged since the last renewal",
|
||
"error_at": "2026-08-27T11:53:46.195553513Z"
|
||
```
|
||
|
||
That is the `default` branch at `cmd/orchestra-worker/main.go:992`.
|
||
|
||
### F21, lifecycle: the phase brief promises a protocol that does not exist
|
||
|
||
`.orchestra/launch.md:62` tells the agent `Orchestra decides when this phase
|
||
ends. Ask for a phase change, do not declare one.` The agent complied and
|
||
printed `Nothing blocks. Ready for a phase change to implement.`
|
||
|
||
```text
|
||
phase brief: "ask for a phase change"
|
||
agent: asks
|
||
worker: has no representation of that request
|
||
coordinator: AdvanceWorkPhase exists, but nothing invokes it autonomously
|
||
```
|
||
|
||
`operations.AdvanceWorkPhase` is reachable only from the HTTP handler at
|
||
`cmd/orchestra/main.go:883` and from `internal/operations/review.go:101`. The
|
||
worker's only post-frame send is a decision notice at
|
||
`cmd/orchestra-worker/main.go:1639`, gated on `len(answer.Decisions) == 0`
|
||
returning early.
|
||
|
||
The fix is a bounded agent intent, not pane-text matching:
|
||
|
||
```go
|
||
type PhaseAdvanceRequest struct {
|
||
From WorkPhase
|
||
To WorkPhase
|
||
}
|
||
```
|
||
|
||
```text
|
||
verified turn boundary
|
||
→ obtain bounded phase intent from harness
|
||
→ validate requested transition
|
||
→ submit to coordinator
|
||
→ operations.AdvanceWorkPhase
|
||
→ if transition requires sealed artifact:
|
||
collect/seal artifact first
|
||
→ rotate/start successor if phase policy requires fresh context
|
||
```
|
||
|
||
`frame` needs no artifact. `research` and `plan` take the same path but must
|
||
seal their artifact before `AdvanceWorkPhase` accepts them.
|
||
|
||
### F22, lifecycle: nothing reacts to `WorkPhaseChanged` at runtime
|
||
|
||
`t.WorkPhase` is read only when a launch context is built, at
|
||
`cmd/orchestra-worker/main.go:369` and
|
||
`internal/orchestrator/orchestrator.go:1218`. No code path rotates, relaunches
|
||
or notifies a live session when the phase changes. A manual advance therefore
|
||
takes effect only at the next launch, and leaves the current idle pane idle.
|
||
|
||
### F23, lifecycle: a decision cannot exist before a submission
|
||
|
||
`answer.Decisions` is the only input `sendPrompt` ever carries post-frame.
|
||
Decisions come from `operations.ReflectSubmission`, and
|
||
`human.PullRequestState.FeedbackAfter` returns input only strictly after a
|
||
submission. A task still in `frame` has no submission, so no decision can be
|
||
recorded for it.
|
||
|
||
Consequence: **F20 cannot be exercised on this task in its current state.** A
|
||
manual phase advance does not produce a post-frame input, because nothing
|
||
reacts to the phase change and no decision can exist yet.
|
||
|
||
### Observation kept separate from F20
|
||
|
||
`❯ go ahead and implement it` appeared in the editor again on attempt 2, cursor
|
||
at `cursor_x=2`, never submitted. The agent transcript at
|
||
`~/.claude/projects/-tmp-test-e2e-worktrees-06G44JZB80MZBEY97196EZN8EC/97b30e4d-90a7-49bc-a9b8-7d9e450831e4.jsonl`
|
||
holds exactly one user message, the launch prompt at 11:55:52.236Z. No worker
|
||
send and no receipt exist for it. Origin unknown, external to Orchestra. F20
|
||
does not absorb it.
|
||
|
||
### Ledger after this checkpoint
|
||
|
||
```
|
||
F6 closed
|
||
F14 closed
|
||
F15 closed by detection, transport fix pending live proof
|
||
F16 closed, both branches live-proven
|
||
F17 fixed, isolated live proof
|
||
F18 open observability
|
||
F19 fixed
|
||
F20 blocked, not merely awaiting: see F23
|
||
F21 open lifecycle, no autonomous phase-advance path
|
||
F22 open lifecycle, no runtime reaction to WorkPhaseChanged
|
||
F23 open lifecycle, no decision path before a submission
|
||
```
|
||
|
||
## F21, F22, F23 implemented, 2026-08-27
|
||
|
||
Run 3 was not shepherded further. Attempt 3 was left unspent: F16 is closed on
|
||
both branches, and another idle phase would have proven nothing new.
|
||
|
||
### F23 was already implemented, and the earlier entry was wrong
|
||
|
||
`human.Reconciler.Reconcile` imports issue comments from `task.Source` and
|
||
records `HumanDecisionRecorded` with no submission involved. It is wired at
|
||
two points in `cmd/orchestra/main.go`: `Store.PreLease`, and
|
||
`Coordinator.ReconcileHumanInput`, which `RemoteTurn` calls at every verified
|
||
turn boundary. `provider.GiteaComments.FetchAfter` reads the task's own issue.
|
||
|
||
The earlier F23 entry traced `EventHumanDecisionRecorded` through
|
||
`internal/operations` only and concluded decisions required a submission. That
|
||
was a scoping error in the search, not a gap in the code. `ReflectSubmission`
|
||
is the pull-request path and stays narrow; it was never the general decision
|
||
source.
|
||
|
||
Consequence for the record: **F20 had a legitimate post-launch send path
|
||
throughout run 3.** A comment on `kami/test-e2e#3` would have produced a
|
||
decision, a `DecisionNotice` at the next boundary, and a receipt.
|
||
|
||
F23 is closed as already-implemented, with tests added for the boundary it
|
||
turns on.
|
||
|
||
### F21, the phase-request protocol
|
||
|
||
The agent asks with a bounded file. Prose is not a protocol, so nothing
|
||
matches on pane text.
|
||
|
||
```
|
||
.orchestra/phase-request.json {"from": "frame", "to": "research"}
|
||
.orchestra/research.json sealed before leaving research
|
||
.orchestra/plan.json sealed before leaving plan
|
||
```
|
||
|
||
At a verified turn boundary the worker validates what it can see locally: the
|
||
phase the agent believes it is in, the legality of the transition, and the
|
||
presence and decode of the artifact the phase must seal. It then calls
|
||
`POST /v1/federation/phase` with the lease epoch and a derived operation id.
|
||
The coordinator calls the existing `operations.AdvanceWorkPhase`.
|
||
|
||
- `internal/operations/workphase.go`: `RequestWorkPhase`, `ErrPhaseRequest`.
|
||
Fences on lease epoch, refuses a stale phase belief, refuses any target but
|
||
the project's next phase, idempotent per operation id.
|
||
- `internal/federation/client.go`: `Client.AdvancePhase`.
|
||
- `cmd/orchestra/main.go`: the `/v1/federation/phase` route.
|
||
- `cmd/orchestra-worker/main.go`: `requestPhase`, `phaseArtifact`,
|
||
`phaseRequestFile`.
|
||
- `internal/agentctx/agentctx.go`: `phaseRequestBrief` renders the protocol
|
||
under the sentence that used to promise it.
|
||
|
||
The operation id is derived, not random: `phase:<task>:<epoch>:<from>:<to>`. A
|
||
redelivery after a lost response carries the same id, so the coordinator
|
||
returns the first event instead of advancing twice.
|
||
|
||
### F22, a phase change ends that cognitive session
|
||
|
||
`herdr.Session` now carries `Phase`, the phase the session was launched to
|
||
run. At a turn boundary a session whose task has moved on is rotated with
|
||
reason `phase_changed`, which is added to the closed handoff-reason set in
|
||
`internal/herdr/adapter.go` and given its own wording in
|
||
`RequestHandoffReason`.
|
||
|
||
One comparison covers both cases: a change this worker requested, and one an
|
||
operator made through `POST /v1/tasks/{id}/phase`. Both leave the same
|
||
evidence, a session built for a phase that is no longer current.
|
||
|
||
### F20 hole found and closed while implementing F22
|
||
|
||
`CLIAdapter.prompt` called `backend.Prompt` and returned. Every rotation and
|
||
handoff prompt therefore went out unconfirmed, which is precisely the failure
|
||
F20 exists to catch. `RequestHandoffReason` is one of its callers, so the
|
||
phase rotation would have inherited it.
|
||
|
||
Fixed at the shared call site rather than per caller: `prompt` now routes
|
||
through `InputConfirmer` and logs the same `input to <pane> confirmed:`
|
||
receipt the worker logs.
|
||
|
||
### Tests
|
||
|
||
`go build ./...`, `go vet ./...` and `go test ./...` all pass.
|
||
|
||
- `internal/operations/phase_request_test.go`: full path with each artifact
|
||
sealed, skipped phase refused, stale phase belief refused, stale epoch
|
||
refused, missing operation id refused, redelivery idempotent, operation id
|
||
recorded.
|
||
- `cmd/orchestra-worker/phase_test.go`: request accepted then session rotates,
|
||
external phase change rotates, unsealed artifact refused locally, malformed
|
||
artifact refused locally, sealed artifact travels with the request, refused
|
||
request retried, decision notice stays undelivered until confirmed.
|
||
- `internal/integration/phase_protocol_test.go`: a pre-submission issue
|
||
comment steers a live leased session and is not re-sent once delivered;
|
||
the phase-request path seals and fences at the coordinator.
|
||
- `internal/human/pullrequest_window_test.go`: pull-request feedback ignores
|
||
anything not strictly after the submission, and applies trust.
|
||
- `internal/agentctx/agentctx_test.go`: the brief names the request file, the
|
||
request shape, and the artifact to seal.
|
||
|
||
### Ledger
|
||
|
||
```
|
||
F6 closed
|
||
F14 closed
|
||
F15 closed by detection, transport fix pending live proof
|
||
F16 closed, both branches live-proven
|
||
F17 fixed, isolated live proof
|
||
F18 open observability, non-blocking
|
||
F19 fixed
|
||
F20 fixed, adapter hole closed; live proof pending run 4
|
||
F21 fixed, tests only
|
||
F22 fixed, tests only
|
||
F23 closed, already implemented; tests added
|
||
```
|
||
|
||
Run 4 starts from here. Both halves must be rebuilt and redeployed before it
|
||
begins, and the deployment boundary recorded as usual.
|
||
|
||
## Deployment boundary for run 4, 2026-08-27
|
||
|
||
Revision `1f5bf7e`, both halves, built from a detached worktree of that commit.
|
||
|
||
| Half | Evidence |
|
||
|---|---|
|
||
| Coordinator, homesrv container | `docker logs orchestra-api`, `orchestra revision 1f5bf7e66e0c6dc8dc1db794ab91193c6c38e118 built 2026-08-27T16:35:37+04:00 dirty false`, `/readyz` 200 |
|
||
| Worker, workpc systemd | pending operator install |
|
||
|
||
Worker sha256 `c188ec78f7b11bd0f3b72da2c3544c5c9477b4505816b30bd0f243f6fa8032da`,
|
||
staged at `~/orchestra-deploy/orchestra-worker.1f5bf7e`.
|
||
|
||
Both worker registrations must report `1f5bf7e` before run 4 starts.
|
||
|
||
### F24, correctness, dormant on the current topology
|
||
|
||
`CLIAdapter.LeasePrompt` (`internal/herdr/adapter.go:161`) calls
|
||
`backend.Prompt` and returns without confirming. It is the last Orchestra-owned
|
||
editor write outside the delivery guarantee.
|
||
|
||
Dormant, not fixed: the coordinator-local harness path cannot execute here. No
|
||
local herdr is eligible, and `Coordinator.adapterFor` refuses a session owned by
|
||
a non-local herdr. Run 4 exercises the federated worker path, whose launch
|
||
delivery is already confirmed at `cmd/orchestra-worker/main.go:454`.
|
||
|
||
The eventual fix:
|
||
|
||
```text
|
||
CLIAdapter.LeasePrompt
|
||
→ backend.Prompt
|
||
→ ConfirmInput
|
||
→ only then acknowledge launch
|
||
```
|
||
|
||
The comment in place there argues that waiting would turn a long first turn
|
||
into a false lease failure. It is obsolete: `ConfirmInput` proves submission,
|
||
not completion of the first turn.
|
||
|
||
Required before claiming coordinator-local harness conformance. Not required
|
||
before this federated burn-in, and changing the revision now would add churn
|
||
run 4 cannot observe.
|
||
|
||
### The y/n approval paths stay separate
|
||
|
||
`internal/herdr/adapter.go:703` and `cmd/orchestra-worker/main.go:1086` write to
|
||
a y/n dialog, not an editor. They need a capture-revision-aware confirmation
|
||
protocol of their own. The editor confirmer would report nonsense on them, so
|
||
they are deliberately outside F20 and stay that way.
|
||
|
||
### Run 3
|
||
|
||
Left running and untouched. Whatever it does from here is additional diagnostic
|
||
evidence. Run 4 does not wait on it.
|
||
|
||
## Run 4: failed conformance, 2026-08-27
|
||
|
||
Task `06G46P6KE25Y04VVF7VRZMHZ78`, issue `kami/test-e2e#4`, revision `1f5bf7e`.
|
||
|
||
```text
|
||
run 4: failed conformance
|
||
|
||
cause:
|
||
f25, claude rotationTick bypassed federatedTurn
|
||
|
||
evidence:
|
||
phase-request.json existed
|
||
no WorkPhaseChanged followed
|
||
no boundary error was emitted
|
||
|
||
additional defects:
|
||
f26 phase brief advertised invalid shortcut
|
||
f27 phase refusal was not delivered to agent
|
||
```
|
||
|
||
### What run 4 did prove
|
||
|
||
The front half of the chain is clean, with no operator lifecycle intervention.
|
||
|
||
| seq | at (UTC) | event |
|
||
|---|---|---|
|
||
| 370 | 13:11:29 | `TaskCreated` v1, 7 acceptance criteria |
|
||
| 371 | 13:11:30 | `TaskLeased` v2, epoch `06G46P6Q4EAPF7DFDBFBTA9RJC` |
|
||
| 372 | 13:11:33 | `TaskLaunchAcknowledged` v3 |
|
||
|
||
Issue to confirmed launch: 4 seconds. F14 passes, against run 1 where ingest
|
||
set no acceptance at all. F15 and F17 pass again with
|
||
`confirmation=editor_cleared submit_attempts=1`.
|
||
|
||
The agent wrote its request 25 seconds after launch:
|
||
|
||
```
|
||
/tmp/test-e2e-worktrees/06G46P6KE25Y04VVF7VRZMHZ78/.orchestra/phase-request.json
|
||
{"from": "frame", "to": "implement"}
|
||
```
|
||
|
||
The artifact half of F21 works. The reading half never ran.
|
||
|
||
### F25, lifecycle: the turn boundary was unreachable on claude
|
||
|
||
`rotationTick` returned early for this harness before reaching the boundary,
|
||
and `federatedTurn` has exactly one call site, below that return.
|
||
|
||
```go
|
||
if w.harness == "claude" {
|
||
if err := w.advanceClaudeContextReset(ctx, id, s); err != nil { ... }
|
||
return
|
||
}
|
||
```
|
||
|
||
Introduced in `7f12c7f`. Consequences on the harness both burn-in runs used:
|
||
no phase request could ever be read, and **every human decision recorded
|
||
against a live claude session went undelivered**.
|
||
|
||
This retracts an earlier claim in this ledger. F23's decisions do reach
|
||
`RemoteTurn`, but on claude the worker never asked, so run 3 had no
|
||
post-launch send path either. The statement that a comment on `#3` would have
|
||
produced a receipt was wrong.
|
||
|
||
Fixed: claude runs the common boundary after its context reset. It still skips
|
||
the occupancy state machine, because it owns its own rollover. A turn boundary
|
||
is not a rotation.
|
||
|
||
### F26, authority: the brief advertised an invalid shortcut
|
||
|
||
The deployed brief rendered `Legal values for "to" from here: research,
|
||
implement`, listing every domain-legal move. The project's path allows only
|
||
`research`. The agent took the shortcut, reasonably.
|
||
|
||
Fixed: the brief names one target and states that a wrong one comes back with
|
||
the right one. The project's path is Orchestra's to know.
|
||
|
||
### F27, lifecycle: a refusal never reached the agent
|
||
|
||
A refused request only reached `w.recordError`. The agent would rewrite the
|
||
same rejected file at every boundary with nothing telling it why, which is the
|
||
silent-loop shape this codebase keeps producing.
|
||
|
||
Fixed: `federation.StatusError` makes a 409 classifiable. A refusal is
|
||
delivered through `sendPrompt`, so it travels under the F20 guarantee, and the
|
||
request file is cleared. A transport failure keeps the file and tells the agent
|
||
nothing, because it is not an answer.
|
||
|
||
### Ledger
|
||
|
||
```
|
||
F6 closed
|
||
F14 closed
|
||
F15 closed by detection, transport fix pending live proof
|
||
F16 closed, both branches live-proven
|
||
F17 fixed, isolated live proof
|
||
F18 open observability, non-blocking
|
||
F19 fixed
|
||
F20 fixed, live proof still pending
|
||
F21 artifact half live-proven, reading half pending
|
||
F22 fixed, live proof pending
|
||
F23 closed as already implemented, undeliverable on claude until F25
|
||
F24 open correctness, dormant on current topology
|
||
F25 fixed, tests only
|
||
F26 fixed, tests only
|
||
F27 fixed, tests only
|
||
```
|
||
|
||
### F28, deployment: the execution runtime shares the worker's cgroup
|
||
|
||
Worker restart destroys agent sessions because the tmux execution runtime
|
||
shares the worker service cgroup. tmux must be independently lifecycle-managed.
|
||
|
||
The worker spawns the server implicitly on its first tmux command, so the
|
||
server and every pane land in `orchestra-worker.service`:
|
||
|
||
```text
|
||
orchestra-worker.service
|
||
├── orchestra-worker
|
||
└── tmux server
|
||
└── claude panes
|
||
```
|
||
|
||
`KillMode` does not fix this. Under `mixed` systemd still sends the final
|
||
SIGKILL to whatever remains in the cgroup once the main process exits, and
|
||
`process` only encodes accidental orphaning, which systemd itself discourages.
|
||
An earlier draft of this entry proposed `mixed` and was wrong.
|
||
|
||
The split:
|
||
|
||
```text
|
||
orchestra-worker.service
|
||
└── orchestra-worker
|
||
|
||
orchestra-tmux.service
|
||
└── tmux -L orchestra
|
||
└── claude sessions
|
||
```
|
||
|
||
`deploy/orchestra-tmux.service` is new. `deploy/orchestra-worker.service` gains
|
||
`After=`/`Wants=` on it, ordering only: a worker that finds the runtime missing
|
||
must report that rather than be stopped by it. The unit carries an idle
|
||
`orchestra-runtime` session so the server outlives its last agent pane.
|
||
|
||
Probed live on a throwaway socket: `tmux -L <s> new-session -d` leaves the
|
||
server running under its own pid after the client exits, so `Type=forking`
|
||
resolves a main pid. `systemd-analyze verify` passes.
|
||
|
||
`User` must match between the two units, because the socket lives under
|
||
`/tmp/tmux-$UID`. The installed worker unit on workpc runs as `kami` while
|
||
`deploy/orchestra-worker.service` still says `orchestra`. The staged copy at
|
||
`~/orchestra-deploy/orchestra-tmux.service` is set to `kami` to match reality.
|
||
|
||
### What F28 retroactively explains
|
||
|
||
- Run 3's `no server running` at 11:02:16 followed the 10:45:26 worker restart.
|
||
- The same at 13:06:22.
|
||
- Run 4's refusal could not be delivered at 13:32:16, seconds after the
|
||
13:31:51 restart.
|
||
- F16's missing-pane branch has been firing on deployment, not on real
|
||
execution-runtime loss. Its proof stands as written, but the trigger was
|
||
self-inflicted.
|
||
|
||
### F28 proof, to run after the split
|
||
|
||
```text
|
||
1. start task and obtain live pane P
|
||
2. record pane/session identity
|
||
3. systemctl restart orchestra-worker
|
||
4. assert pane P still exists with identical agent identity
|
||
5. new worker starts
|
||
6. worker reconciles lease + pane P
|
||
7. no TaskReleased / new lease caused merely by worker restart
|
||
8. agent continues under the same lease epoch
|
||
```
|
||
|
||
Then separately, to restore the meaning of F16's missing-pane branch:
|
||
|
||
```text
|
||
restart orchestra-tmux
|
||
→ pane disappears
|
||
→ F16 refuses renewal
|
||
→ lease expires and requeues
|
||
```
|
||
|
||
### Run 4 diagnostic continuation, partial
|
||
|
||
The chain ran and failed only at the last step, at 13:32:16:
|
||
|
||
```
|
||
deliver phase refusal 06G46P6KE25Y04VVF7VRZMHZ78: tmux send-keys ... -l -- Orchestra refused your phase request: phase request refused: task 06G46P6KE25Y04VVF7VRZMHZ78 may only move to "research", not "implement"
|
||
...: no server running on /tmp/tmux-1000/orchestra: exit status 1
|
||
```
|
||
|
||
Live-proven by that one line: F25, the boundary ran on claude at all; F21's
|
||
reading half; F26's premise, the coordinator naming the one valid target; F27,
|
||
the 409 classified and routed to `sendPrompt`. The request file survived the
|
||
failed send, which is the correct branch.
|
||
|
||
Not proven: the receipt itself, because F28 had already destroyed the pane.
|
||
|
||
## F28 closed, and three defects it uncovered, 2026-08-28
|
||
|
||
The unit split is installed on workpc and F28 is proven live in both
|
||
directions. Three new defects, F29, F30 and F31, were found while setting
|
||
the proof up. None was visible in tests.
|
||
|
||
### The tmux unit, as installed
|
||
|
||
The staged `Type=forking` draft was wrong in a way the earlier entry did not
|
||
catch: without `PIDFile=`, `systemctl show orchestra-tmux -p MainPID` returned
|
||
`0`. Systemd had no main process to watch, so a crashed server would read as a
|
||
clean exit and `Restart=on-failure` would never fire.
|
||
|
||
`tmux -D` runs the server in the foreground, which gives systemd the real pid.
|
||
It also turns `exit-empty` off, so the synthetic `orchestra-runtime` session is
|
||
no longer needed to hold the server open past its last agent pane. Nothing in
|
||
the tree referenced that session.
|
||
|
||
```ini
|
||
[Service]
|
||
Type=simple
|
||
User=kami
|
||
ExecStart=/usr/bin/tmux -D -L orchestra
|
||
Restart=on-failure
|
||
RestartSec=1
|
||
```
|
||
|
||
Verified after install: `MainPID=3073848`, cgroup holds only the server, and
|
||
the server stays alive with zero sessions.
|
||
|
||
`deploy/orchestra-worker.service` said `User=orchestra` while the installed
|
||
unit ran as `kami`. Since the socket is `/tmp/tmux-$UID`, that mismatch would
|
||
have pointed the worker at a different server. Corrected in `fc4c29f`.
|
||
|
||
### F28 proof, live
|
||
|
||
Vehicle: disposable task `06G474FXXFW7V54RGF70ZS3SRG`, forced onto
|
||
`workpc-claude`, leased at 20:05:07Z under epoch `06G49MW2CHNVT01SHEKVYJ2170`,
|
||
pane `orchestra-06g474fxxfw7v54rgf70zs3srg-59bf1ee9:1.0`, pane pid `3012539`.
|
||
|
||
| Assertion | Result |
|
||
|---|---|
|
||
| Pane survives worker restart | 3 restarts, pid `3012539` every time |
|
||
| Same agent identity | unchanged |
|
||
| Same lease epoch continues | `06G49MW2CHNVT01SHEKVYJ2170` |
|
||
| Worker reconciles lease and pane | `leases` and `sessions` rehydrated each time |
|
||
| No `TaskReleased` from a restart | event log empty after seq 385 |
|
||
| No replacement `TaskLeased` | same |
|
||
| Subsequent renewal succeeds | 00:25:11 local, v6 → v7, `progress_sha` recorded |
|
||
|
||
The event log is the load-bearing evidence. The only `TaskReleased` on this
|
||
task is seq 382 at 20:04:06Z, from an earlier failed launch, well before the
|
||
first restart at 20:08:33Z. The restarts appended nothing at all.
|
||
|
||
Forcing a renewal by shortening the worker's cached `until` does not work. The
|
||
worker re-hydrates the authoritative deadline from the coordinator on startup,
|
||
before any renewal tick runs. The cache is not a lever.
|
||
|
||
### Forcing a task onto one harness
|
||
|
||
`config.jsonc` gave `workpc-claude` the capabilities `["code", "review"]`, and
|
||
`workpc-opencode` `["code", "review", "opencode"]`. No capability selected
|
||
claude uniquely, so an operator test could not pin placement. `claude` was
|
||
added to `workpc-claude`, mirroring the convention opencode already followed.
|
||
Router rejections at `/v1/router/health` named the gate directly, which is what
|
||
made this a two-minute diagnosis instead of an afternoon.
|
||
|
||
### F29: the browser could not act on a leased task
|
||
|
||
`validateTransition` fences `TaskReleased`, `TaskBlocked` and `TaskCompleted`
|
||
on a leased task against the live `harness_id` and `lease_epoch`. The UI action
|
||
handler sent neither, so all three returned 409 on exactly the tasks the UI
|
||
listed them as enabled for.
|
||
|
||
Found by trying to park run 4: eight block attempts, stable v8, all 409 `task
|
||
version conflict`. The workaround was to block during the queued window, where
|
||
`t.Lease` is nil and the fence does not apply.
|
||
|
||
Fixed in `efd0a5e`. The fence exists to reject a stale writer, not the
|
||
operator, so the handler now carries the lease it just read. The version CAS on
|
||
the append still rejects a racing write. A missing request body also wrote into
|
||
a nil map, which is fixed alongside.
|
||
|
||
### F30: an unpushed release pinned the worker's only slot
|
||
|
||
A release transaction that never reached `anchor_pushed` has no artifact.
|
||
`tx.Ref` is empty and no successor can pick anything up. The event handler kept
|
||
its session mapping alive anyway, so once the pane was gone the mapping was
|
||
immortal. `health()` reports `ActiveTask` straight out of `w.sessions`, so the
|
||
coordinator saw the harness as permanently busy and never leased to it again.
|
||
|
||
Live on `workpc-claude`, stuck at phase `prepared` behind this error:
|
||
|
||
```
|
||
adapter: invalid handoff answer: invalid handoff authored field: prose smuggled into list
|
||
```
|
||
|
||
It produced no log line for fifteen minutes. Freeing it needed hand surgery on
|
||
`/var/lib/orchestra/worker-state/workpc-claude.json` with the worker stopped.
|
||
|
||
Fixed in `2dc90bd`. The mapping is kept only while an anchor actually exists,
|
||
and the transaction is dropped with it, since nothing can advance it.
|
||
|
||
Worth recording separately: the rejected handoff artifact came from an agent
|
||
running the operator's direct-prose stop hook. That is a live interaction
|
||
between an operator hook and the handoff contract, not a defect in either.
|
||
|
||
### Ledger
|
||
|
||
```
|
||
F6 closed
|
||
F14 closed
|
||
F15 closed by detection, transport fix pending live proof
|
||
F16 closed, both branches live-proven
|
||
F17 fixed, isolated live proof
|
||
F18 open observability, non-blocking, cost diagnosis time twice this session
|
||
F19 fixed
|
||
F20 fixed, live receipt still pending
|
||
F21 reading and artifact halves live-proven; accepted transition pending
|
||
F22 fixed, live proof pending
|
||
F23 closed as already implemented
|
||
F24 open correctness, dormant on current topology
|
||
F25 fixed, live-proven
|
||
F26 fixed, live-proven
|
||
F27 fixed, live-proven
|
||
F28 CLOSED, live-proven in both directions
|
||
F29 fixed in efd0a5e, needs a coordinator rebuild
|
||
F30 fixed in 2dc90bd, needs a worker rebuild
|
||
F31 fixed in 6523002, needs a worker rebuild, BLOCKS every rotation
|
||
```
|
||
|
||
### Deployment state
|
||
|
||
Both fixes are committed and unbuilt. The running coordinator and worker are
|
||
still `fbaaf79`. Run 5 should start from a deployment carrying `efd0a5e` and
|
||
`2dc90bd`, or it will hit F30 again the first time a handoff artifact is
|
||
rejected.
|
||
|
||
### F28 runtime-death direction, live
|
||
|
||
`systemctl stop orchestra-tmux` at 00:39, restarted at 00:39:37 as MainPID
|
||
`3073848`. Every pane died with the old server.
|
||
|
||
```text
|
||
20:45:07Z renewal window opens, no TaskLeaseRenewed appended F16 refused
|
||
20:55:07Z lease deadline
|
||
20:55:50Z state queued v8, lifecycle reclaimed, failure_class lease_expired
|
||
```
|
||
|
||
Runtime death is now distinct from worker deployment. F16's missing-pane branch
|
||
fires on real execution loss, which is what it was written for.
|
||
|
||
The same window reproduced F30 a second time, on the deployed `fbaaf79` worker
|
||
that lacks `2dc90bd`: session still pinned, release transaction stuck at
|
||
`prepared`, same rejected handoff artifact.
|
||
|
||
### F31: every rotation failed on a length check that named the wrong cause
|
||
|
||
Two different tasks, two different agents, one error:
|
||
|
||
```
|
||
adapter: invalid handoff answer: invalid handoff authored field: prose smuggled into list
|
||
```
|
||
|
||
`parseHandoffAnswer` builds `Action` by joining the agent's `NEXT` and `WHY`
|
||
answers with `" — "`. `Validate` then held that join to `maxAuthoredLine`, 200
|
||
characters. The prompt asks for a sentence each and names no budget, so two
|
||
ordinary sentences do not fit.
|
||
|
||
The message named a branch the answer cannot reach. `parseHandoffAnswer` splits
|
||
on newlines and trims every field, so no authored field ever contains `"\n#"`.
|
||
Length was the only reachable cause, and the agent was never told what to
|
||
shorten. It could not self-correct, so the same text failed on every retry.
|
||
|
||
This is why run 4 and the F28 disposable task both died at rotation, and why
|
||
each left the stuck transaction behind that pinned the worker slot.
|
||
|
||
Fixed in `6523002`: `Action` carries the budget of both lines, the error names
|
||
the actual length and limit, and the prompt states the limit.
|
||
|
||
F31 blocks run 5 outright. Run 5's chain is four rotations.
|
||
|
||
## Deployment of 93338b7, and F32, 2026-08-28
|
||
|
||
Both halves deployed from one revision, built from a throwaway worktree because
|
||
the shared checkout still carries 20 uncommitted paths from another session.
|
||
|
||
```text
|
||
coordinator 93338b72cec5facdb991d5dfa56eb9d3195749d0 image sha256:8b57491da293 readyz 200
|
||
worker 93338b72cec5facdb991d5dfa56eb9d3195749d0 sha256 168bf20724ec7e03bb5bd19320d4b698d31441d15ce3706d0498d1c85ebd044f
|
||
tmux runtime MainPID 3073848, untouched across the worker install
|
||
```
|
||
|
||
The worker install is itself further F28 evidence. A real deployment left the
|
||
runtime and its cgroup alone.
|
||
|
||
### F29, live-proven
|
||
|
||
Blocking the F28 disposable task while leased returned `HTTP 200` and moved it
|
||
to `blocked v10`. The same call returned 409 eight times before this deploy.
|
||
|
||
### F30, live-proven, and the defect hiding behind it
|
||
|
||
The stuck `releases` entry cleared on the block, which is the fix working. The
|
||
session did not clear. It was marked `quarantined` instead, because `Kill`
|
||
returned an error for a pane that no longer exists.
|
||
|
||
### F32: an empty tmux server answers differently
|
||
|
||
```text
|
||
tmux has-session -t '=<gone>' → "no current target" exit 1
|
||
```
|
||
|
||
`hasSession` recognised `can't find session`, `no server running`, `no sessions`
|
||
and a socket-missing case. It did not recognise `no current target`, so a
|
||
missing session read as a real error.
|
||
|
||
This is a side effect of `dea56e4`. Before the runtime became its own unit, the
|
||
server exited with its last session and answered `no server running`, which was
|
||
already handled. `tmux -D` keeps the server alive past its last pane, so this
|
||
reply had never been produced before today.
|
||
|
||
Consequences: `Kill` fails for an already-dead pane, the quarantine never
|
||
clears, and the worker's only session slot stays pinned. `AgentStatus` shares
|
||
the helper and errored where it should report `exited`, which means F16's
|
||
missing-pane branch was refusing renewal by error rather than by detection.
|
||
|
||
Fixed in `92adf04`.
|
||
|
||
### Ledger
|
||
|
||
```
|
||
F28 closed, live-proven in both directions
|
||
F29 fixed in efd0a5e, DEPLOYED, live-proven
|
||
F30 fixed in 2dc90bd, DEPLOYED, partially live-proven, rest blocked by F32
|
||
F31 fixed in 6523002, DEPLOYED, awaiting run 5's first rotation
|
||
F32 fixed in 92adf04, needs a rebuild, BLOCKS leasing on workpc-claude
|
||
```
|
||
|
||
## Run 5, first attempt: the launch never went off, 2026-08-28
|
||
|
||
Filed as `kami/test-e2e#5`, ingested as `06G4A4F0TFXKZHJE48N05XN1HG`, placed on
|
||
`workpc-claude` by normal routing. Issue filed to confirmed launch: 44 seconds.
|
||
|
||
Then nothing moved for seven minutes. The pane held the launch text unsent:
|
||
|
||
```text
|
||
❯ @.orchestra/launch.md is your complete Orchestra launch instruction. Read
|
||
.orchestra/launch.md now and follow it.
|
||
|
||
0 tokens ⏱️ 0m 0s session ... - none
|
||
```
|
||
|
||
### F33: an editor that never rendered confirmed the submit
|
||
|
||
```text
|
||
first_submit_at 21:13:19.570302125Z
|
||
confirmed_at 21:13:19.577191672Z
|
||
confirmation editor_cleared submit_attempts=1
|
||
```
|
||
|
||
Seven milliseconds. `ConfirmInput`'s first poll ran before the TUI rendered the
|
||
pasted text, found an empty editor, fell through to the idle branch, and called
|
||
that proof of submission.
|
||
|
||
The two launches that worked earlier took `submit_attempts=2` over roughly
|
||
500ms. The difference between a good launch and this one was scheduling luck,
|
||
which is why it had never been seen before.
|
||
|
||
The cost is worse than a failed launch. The coordinator believes the agent is
|
||
working, so the lease renews against a pane that will never produce anything,
|
||
and the failure only surfaces at expiry half an hour later.
|
||
|
||
Fixed in `32220d9`. An empty editor counts as proof only after it has held the
|
||
text, or after it has stayed empty past a settle window. Text seen and then
|
||
gone still confirms immediately, and so do `busy`, `blocked` and `queued`. With
|
||
the fix the failing case reaches the existing resubmit path instead: the text
|
||
appears, is recognised as unsubmitted, and Enter is resent.
|
||
|
||
### Recovery, deliberately not by hand
|
||
|
||
Pressing Enter in the pane would fix this run and prove nothing. The lease is
|
||
left to expire and relaunch on its own, which is the autonomous path run 3
|
||
already proved. The worker carrying `32220d9` is deployed first, so the
|
||
relaunch uses the fixed confirmation.
|
||
|
||
### Ledger
|
||
|
||
```
|
||
F29 DEPLOYED, live-proven
|
||
F30 DEPLOYED, live-proven
|
||
F31 DEPLOYED, still awaiting a first rotation
|
||
F32 DEPLOYED, live-proven, slot reaped and freed
|
||
F33 fixed in 32220d9, needs a rebuild, BLOCKS every launch nondeterministically
|
||
```
|
||
|
||
## Run 5, second attempt: two more layers, 2026-08-28
|
||
|
||
The stuck launch was recovered by hand, one Enter into the pane. That is an
|
||
operator recovery of a keystroke Orchestra believed it had already delivered,
|
||
not a lifecycle intervention, and F35 turns it into an action for next time.
|
||
|
||
### F34, live-proven while being written
|
||
|
||
```text
|
||
01:33:37 first renewal allowed, no baseline, lease extends 21:43 -> 22:03
|
||
```
|
||
|
||
The pane had not changed since 01:13 and held unsent text. The renewal gate
|
||
exempts a lease with no baseline, which grants a full free period to exactly
|
||
the case the gate exists to catch. Fixed in `92f32d6` by baselining the
|
||
progress hash at launch.
|
||
|
||
### F35: nothing could re-poke a pane
|
||
|
||
The UI offered `grant_approval`, `deny_approval`, `handoff`, `release`, `block`
|
||
and `complete`. Orchestra can put text in an editor and be wrong about whether
|
||
it landed, and the only recovery was to destroy the lease and wait out expiry.
|
||
|
||
`resubmit` presses Enter on text Orchestra itself submitted. No lifecycle
|
||
change, so no event; fenced like an approval, on a live worker-owned pane at
|
||
the capture revision the operator was looking at. In `92f32d6`.
|
||
|
||
### The gate, half cleared
|
||
|
||
```text
|
||
01:36:35 phase request accepted: frame to research
|
||
01:36:35 acceptance delivered, confirmation=queued
|
||
01:36:35 phase changed: session rotating
|
||
```
|
||
|
||
Further than any previous run. `work_phase` reached `research` and the request
|
||
artifact was consumed.
|
||
|
||
### F36: the two halves of the handoff contract disagreed
|
||
|
||
```text
|
||
before adapter: invalid handoff answer: invalid handoff authored field: prose smuggled into list
|
||
after adapter: upload handoff: invalid handoff meta
|
||
```
|
||
|
||
F31 cleared the parse, and the release then failed one stage later.
|
||
`handoffReason` has emitted `phase_changed` since phase rotations landed. The
|
||
validator's `reasons` list was never extended, so every phase rotation built a
|
||
handoff it then refused.
|
||
|
||
Fixed in `b9ca365`. The test asserts the property, not the constant: every
|
||
reason the adapter can produce must survive `Validate`, including its fallback.
|
||
|
||
### Pattern worth naming
|
||
|
||
Six defects tonight, each hidden behind the one before it. F30 was reachable
|
||
only once F29 let an operator act, F32 only once F30 tried to reap, F34 and F36
|
||
only once F33 and F31 let execution get that far. Tests passed throughout. The
|
||
only thing that surfaced any of them was running the system.
|
||
|
||
### F37: the handoff carried the write Orchestra asked for
|
||
|
||
One stage past F36:
|
||
|
||
```text
|
||
adapter: upload handoff: invalid handoff command: must not point to a handoff or report
|
||
```
|
||
|
||
`Handoff.Command` is the last command observed in the pane. The handoff prompt
|
||
tells the agent to write `.orchestra-handoff-report.md` and stop, so that write
|
||
is almost always the last command there. `.orchestra-handoff-report.md` matches
|
||
every alternative in `circularCommand`, so every rotation failed on Orchestra's
|
||
own instruction.
|
||
|
||
Fixed in `cb7782d`. `lastObservedCommand` skips commands the validator would
|
||
call circular. The rule is asked for through `continuity.IsCircularCommand`
|
||
rather than restated, because restating a rule in two places is exactly what
|
||
produced F36.
|
||
|
||
## Run 5: the continuity chain, live and autonomous, 2026-08-28 01:47
|
||
|
||
The first successful phase rotation in this project's burn-in history.
|
||
|
||
```text
|
||
handoff_ref abe51f5c663f4d59bba2c98c0418554278ec2a9b75bfc504bd36757666e3948a
|
||
transaction_id 06G4A9VSBE5N5BF8BE3J0PSAER the one that had stuck at "prepared"
|
||
anchor_sha c643aaab87b4b1f79b9f1d909bf8788a0e25fe61
|
||
pickup_acknowledged true
|
||
new epoch 06G4AC5KX225JGXAEWH52NNPB0
|
||
new pane created 01:47:09
|
||
session phase research
|
||
task leased v9, work_phase research, lifecycle started
|
||
```
|
||
|
||
| Step | Evidence |
|
||
|---|---|
|
||
| Handoff answer parses | F31, no `invalid handoff answer` |
|
||
| Handoff validates | F36 and F37, `handoff_ref` exists |
|
||
| Anchor pushed | `anchor_sha c643aaab` |
|
||
| Release commits | transaction gone from `releases` |
|
||
| Predecessor slot frees | old session dropped |
|
||
| Successor leases and starts | new epoch, new pane, phase `research` |
|
||
| Pickup validates | `pickup_acknowledged: true` |
|
||
|
||
No lifecycle shepherding. The only operator action in this run was one Enter
|
||
into a pane holding unsent input, which F35 turns into an action.
|
||
|
||
Two fixes proved themselves incidentally in the same event. The relaunch logged
|
||
`submit_attempts=2`, so F33's resend path does real recovery rather than
|
||
existing only in tests. The new lease carries `progress_sha` at once, so F34's
|
||
launch baseline survives into the real worker path.
|
||
|
||
### Remaining gates
|
||
|
||
```text
|
||
research -> research.json sealed -> request -> rotation -> plan launch carries accepted research
|
||
plan live -> correction comment -> HumanDecisionRecorded -> RemoteTurn -> DecisionNotice receipt, once
|
||
plan.json sealed -> request -> rotation
|
||
implement launch -> correction above accepted plan above accepted research
|
||
```
|
||
|
||
The correction is held until the plan successor is definitely live. Posting
|
||
earlier still exercises F20 and F23, but it cannot prove a mid-plan correction
|
||
outranks accepted research and the current trajectory.
|
||
|
||
### F38 and F39: the agent guessed a schema, and was never told it guessed wrong
|
||
|
||
Run 5 sealed `research.json` at 01:47:52 and wrote
|
||
`{"from": "research", "to": "plan"}`. Nothing moved for five minutes. The
|
||
operator noticed before the system reported anything.
|
||
|
||
```text
|
||
research artifact: json: cannot unmarshal string into Go struct
|
||
field Research.dead_ends of type workphase.DeadEnd
|
||
```
|
||
|
||
F38: the brief said `Seal .orchestra/research.json before you ask` and
|
||
described the contents in prose. It never stated the schema. The agent wrote
|
||
`dead_ends` as strings where the decoder wants `{tried, why_failed}` objects.
|
||
Everything else in the artifact was correct.
|
||
|
||
F39: `requestPhase` refused through `recordError` alone. `answerRefusedPhase`
|
||
ran only on a coordinator 409, so a local decode failure reached nobody. The
|
||
session parked at a boundary with no feedback, and worker health holds one
|
||
error slot (F18), so even that trace was overwritable.
|
||
|
||
F39 is the serious half. F38 alone costs one corrected file. F39 turns any
|
||
local refusal into a silent stall.
|
||
|
||
Fixed in `0bd86e2`. The brief carries the shape, and every local refusal now
|
||
reaches the agent. `TestPhaseSealSchemasDecode` decodes each documented shape
|
||
with the function the worker uses: flipping `dead_ends` back to strings
|
||
reproduces run 5's exact error at build time.
|
||
|
||
## Run 5: the whole chain, live, 2026-08-28 02:05
|
||
|
||
Filed as `kami/test-e2e#5`, task `06G4A4F0TFXKZHJE48N05XN1HG`, harness
|
||
`workpc-claude`, both halves on `cd1675a`.
|
||
|
||
```text
|
||
01:13:19 launched, then stalled: F33 confirmed an editor that had not rendered
|
||
02:00:05 worker restarted on cd1675a
|
||
02:00:06 phase request refused, delivered into the pane (F39)
|
||
02:02 agent rewrote research.json, unprompted by any human
|
||
02:03:00 research to plan accepted
|
||
02:03:31 plan session launched carrying accepted research
|
||
02:04:05 plan.json refused: changes[0].intent 840 characters exceeds the 500 bound
|
||
02:04:07 operator correction posted to the issue
|
||
02:04:10 delivered to the live plan session, decision 06G4AG40NR866DVD875ERFZT70
|
||
02:04:30 plan to implement accepted
|
||
02:04:37 plan.json names it: "Human decision: JSON keys must be snake_case..."
|
||
02:05:06 implement session launched
|
||
```
|
||
|
||
Three seconds from comment to delivery. `delivered_decisions` holds exactly one
|
||
id, so the notice did not repeat.
|
||
|
||
### The authority order, as rendered
|
||
|
||
```text
|
||
## Goal
|
||
## Acceptance
|
||
## Current human decisions
|
||
- correction (operator_instruction): for --json, use snake_case keys.
|
||
keep the default text output byte-for-byte unchanged.
|
||
## Current phase implement
|
||
## Repository rules
|
||
## Verified git state
|
||
## Accepted research
|
||
## Accepted plan
|
||
## Continuity from the previous session
|
||
### Dead ends already tried
|
||
```
|
||
|
||
Human correction above accepted plan, above accepted research, stale continuity
|
||
below all of them. That is the target chain, live.
|
||
|
||
### What each fix bought, observed rather than argued
|
||
|
||
- F33: the 02:03:31 and 02:05:06 launches both logged `submit_attempts=2`. The
|
||
resend path is doing real recovery on every rotation.
|
||
- F39: two refusals delivered, one for `research.json` and one for `plan.json`.
|
||
The agent corrected both without a human touching anything. Before tonight
|
||
each would have been a silent stall until lease expiry.
|
||
- F38: the plan brief carried the plan schema, so the plan artifact failed on a
|
||
length bound rather than on a guessed shape.
|
||
- F20 and F23: one comment, one decision, one delivery, one receipt.
|
||
|
||
### Standing caveat
|
||
|
||
Run 5 is implementing, not finished. The chain this run set out to prove is
|
||
proven. Completion, the review phase, and the final report are still ahead.
|
||
|
||
### F40: the terminal phase never learned how to finish
|
||
|
||
Run 5's review agent finished at 02:12 and stopped. No `phase-request.json`, no
|
||
handoff, no `.orchestra/done`, and nothing in the worker journal after 02:06:21.
|
||
|
||
`legalPhaseTransitions` makes review terminal: its only move is backwards to
|
||
`implement`, for rework. A review that passes has nothing to ask for.
|
||
|
||
The worker has always finalised on `.orchestra/done`
|
||
(`cmd/orchestra-worker/main.go:546`). No brief ever named that file. Grepping a
|
||
rendered `launch.md` for it returned zero matches, in every phase.
|
||
|
||
So the completion signal existed on one side of the contract only. An agent
|
||
whose review passed had no instruction at all, and stopping was the correct
|
||
reading of what it had been told.
|
||
|
||
Fixed in the phase brief: the terminal phase now states the marker, when to
|
||
write it, and that it is exclusive with asking to go back.
|
||
`TestTerminalPhaseNamesTheCompletionSignal` also asserts the negative, so a
|
||
phase that can still ask is never told to finish instead.
|
||
|
||
### F41: harness chrome counted as agent progress
|
||
|
||
Run 5's review agent produced nothing after 02:12. The 02:46 renewal was
|
||
granted anyway, and the lease moved to 03:16. Each renewal window bought
|
||
another 30 minutes, so recovery by expiry had no bound.
|
||
|
||
Reproduced the worker's stored `progress_sha` byte for byte from the live pane,
|
||
so the branch taken at `cmd/orchestra-worker/main.go:1020` was
|
||
`progress != l.ProgressSHA`. Not `IsBusy`: the 200-line capture held zero
|
||
`esc to interrupt` markers, so `AgentStatus` returned `idle`. Not an empty
|
||
baseline either: it held `3d93ca89`.
|
||
|
||
`PaneProgress` dropped prompt lines only, so Claude Code's status footer stayed
|
||
in the digest. The mutable fields there are the rolling usage percentage, the
|
||
context counter, and the version notice. A 180-second sample showed no churn, so
|
||
this leaked intermittently rather than constantly. One tick inside a window was
|
||
enough.
|
||
|
||
Fixed in `2417a39`. `PaneProgress` cuts from the editor's lower rule and then
|
||
trims the spinner summary and version notice above it. `AgentStatus` still reads
|
||
the raw capture, so the busy markers living in the footer are unaffected.
|
||
|
||
Proven live in both directions on the first window after deployment: the
|
||
03:13:48 renewal took version 28, the 03:16:13 window was refused, the lease
|
||
expired at 03:18:43, and review relaunched at 03:20:14 with the F40 brief. The
|
||
relaunched pane then showed a new footer field, `5h: 15%`, which the old digest
|
||
would have counted as work.
|
||
|
||
Lease TTL moved to 5 minutes in the same commit, from `domain.LeaseTTL`, with
|
||
renewal at half of it. Reclaiming a stalled pane happens only at expiry.
|
||
|
||
### F42: the result commit refused the completion marker
|
||
|
||
The relaunched review agent wrote `.orchestra/done` at 03:20:48, 34 seconds
|
||
after launch. The worker recognised it, confirmed the agent idle, and then
|
||
failed the result commit once every five seconds:
|
||
|
||
```text
|
||
stage result: The following paths are ignored by one of your .gitignore files:
|
||
.orchestra/done
|
||
```
|
||
|
||
`internal/herdr/adapter.go:132` writes `.orchestra/.gitignore` containing `*`,
|
||
so the marker is ignored. The staging step passed `:!.orchestra/done`, and git
|
||
refuses an add whose pathspec names an ignored path. Reproduced in a scratch
|
||
repo on git 2.55.0, both the failing and the working form.
|
||
|
||
Fixed in `dcd9af4`: the exclusion names the directory, `:!.orchestra`. That
|
||
holds whether or not the inner `.gitignore` exists.
|
||
|
||
The failure retried for 22 minutes with the task stuck in review and nothing
|
||
observable outside the journal, because each identical error overwrote the
|
||
single `last_error` slot. F18, again, and the second time in one night that it
|
||
turned a five-second loop into archaeology.
|
||
|
||
### Run 5 reached TaskCompleted
|
||
|
||
On the first tick after the `dcd9af4` restart at 06:47:50:
|
||
|
||
- task state `completed`, lease released, `.orchestra/done` removed
|
||
- pane closed, no `orchestra-*` sessions left on the tmux socket
|
||
- result branch pushed:
|
||
`refs/heads/orchestra/scratch/orchestra-06g4a4f0tfxkzhje48n05xn1hg-73a2e4fd`
|
||
at `fb15c61`
|
||
|
||
Two links of the tail did not run, and neither is a defect. The deployed
|
||
`test-e2e` project declares only `id`, `machine_affinity`, `repo` and
|
||
`worktree_root`, so no quality gate exists to run. The deployed `config.jsonc`
|
||
has no `sources` and no `delivery` keys, so no PR could be opened.
|
||
|
||
Proven chain: review, done recognised, finalise, result branch pushed,
|
||
`TaskCompleted`. Submission, human PR review and merge remain unproven, and they
|
||
need registry configuration rather than code.
|
||
|
||
## Run 6, 2026-08-28: the PR tail, and the baseline conformance run
|
||
|
||
The last handoff said submission, human review and merge needed configuration
|
||
rather than code. That was wrong. The whole event log held zero `ReviewRecorded`
|
||
and zero `TaskSubmitted`: nothing in the running system ever called either
|
||
endpoint, so the publisher, the merge reflection and the human trust boundary
|
||
had no entry point at all. Four links were missing, and three more defects
|
||
surfaced while proving them.
|
||
|
||
### F43, the reviewed change had no path to the human
|
||
|
||
The worker's completion tail called `federation.Client.Complete` directly and
|
||
emitted `TaskCompleted`. `POST /v1/tasks/{id}/review` and
|
||
`POST /v1/tasks/{id}/submission` existed and had no live caller; `task pr` is
|
||
named in six documents and implemented nowhere. `GiteaPublisher.Push` also ran
|
||
`git push` from `projectRoots[project]/<task>` on the coordinator, a directory
|
||
that does not exist for a worker-owned project whose worktree is on workpc.
|
||
|
||
Fixed in `e8d04d7`:
|
||
|
||
- `finalize` commits first and runs the gate against the committed tree, so
|
||
`GateResult.SHA` is the commit being submitted. `CheckSubmission` requires
|
||
gate sha, review sha and head sha to be one commit, which a gate run on the
|
||
pre-commit tree can never satisfy.
|
||
- `POST /v1/federation/workers/<id>/submit` seals the review and submits. A
|
||
blocking review returns the task to implementation; a project with no forge
|
||
still completes directly.
|
||
- The reviewer's brief names `.orchestra/review.json` and no longer claims a
|
||
diff is supplied. Nothing populates `agentctx.Evidence` on the federated
|
||
path, so the brief described material the session never received.
|
||
- `Push` asks the forge what the branch holds before reaching for a checkout.
|
||
|
||
### F44, the Gitea source ingested Orchestra's own pull requests
|
||
|
||
The first submission this deployment ever made, `kami/test-e2e#8`, came back one
|
||
minute later as task `06G4E83E4KRXM8DS90M2648MGM` with the submission packet as
|
||
its description. That task would have implemented, reviewed and submitted again,
|
||
opening a pull request per cycle. The issues endpoint returns pull requests and
|
||
nothing filtered them. The webhook had the same hole from the other side: a
|
||
`pull_request` delivery leaves the `issue` key empty and would have appended a
|
||
task numbered 0 with no title.
|
||
|
||
Contained by blocking the spawned task, fixed in `6ccc755`.
|
||
|
||
### F45, the submission reflection loop was dead code
|
||
|
||
`if len(pullRequests) > 0` guarded the loop 600 lines before the Gitea wiring
|
||
that writes to that map. The length was always zero, the goroutine never
|
||
started, and a merged pull request could never complete its task. Live: a
|
||
trusted comment on PR #8 moved nothing until `4af9880` moved the block below
|
||
the wiring.
|
||
|
||
### F46, stale review findings survive a changes-requested round trip
|
||
|
||
The worktree is not recreated, so `.orchestra/review.json` from the first review
|
||
is still present when the second one starts. A reviewer that writes
|
||
`.orchestra/done` without rewriting it would have the earlier findings sealed
|
||
against the new commit, and a stale pass is indistinguishable from a fresh one.
|
||
Observed at 10:57:48 with a file from 10:55:45. That reviewer did rewrite it, so
|
||
the run stands. Fixed in `063a3ab`.
|
||
|
||
### Baseline conformance run
|
||
|
||
Task `06G4E6F69AKP2PA00S28D7ASBC`, issue `kami/test-e2e#7`, `workpc-claude`,
|
||
tmux backend. Coordinator `4af9880`, worker `e8d04d7`. No manual lifecycle
|
||
intervention: every event below was emitted by the plane.
|
||
|
||
```text
|
||
06:41:14 TaskCreated issue #7 polled
|
||
06:41:37 WorkPhaseChanged frame -> research
|
||
06:42:57 WorkPhaseChanged research -> plan
|
||
06:45:12 WorkPhaseChanged plan -> implement
|
||
06:46:27 WorkPhaseChanged implement -> review
|
||
06:47:48 ReviewRecorded blocking 0, sha 41dee802
|
||
06:47:49 TaskSubmitted PR 8 opened, sha 41dee802
|
||
06:55:11 TaskChangesRequested trusted comment by kami
|
||
06:55:11 WorkPhaseChanged review -> implement
|
||
06:57:47 WorkPhaseChanged implement -> review
|
||
07:00:09 ReviewRecorded blocking 0, sha 0d1585df
|
||
07:00:09 TaskSubmitted PR 8 reused, sha 0d1585df
|
||
07:02:07 merge by kami
|
||
07:02:11 TaskCompleted receipt binds merge f48e363d
|
||
```
|
||
|
||
Acceptance, each checked against the artifact rather than the log line:
|
||
|
||
- The gate ran against the exact submitted commit. Gate artifact
|
||
`bd3549d6`: `{"command":"bash -n scripts/*.sh && bash
|
||
scripts/orchestra_e2e_healthcheck.sh","exit_code":0,"sha":"0d1585df"}`.
|
||
- `TaskSubmitted` binds all five: gate sha, review sha, result sha, remote ref
|
||
`origin/orchestra/06G4E6F69AKP2PA00S28D7ASBC`, and PR 8. The PR head reads
|
||
`0d1585df` at the forge.
|
||
- Resubmission reused PR 8 with a new gate ref and a new review ref.
|
||
- A trusted human reopened the task and the change reached merged `master`:
|
||
`--quiet` is documented in `--help`, which is what the comment asked for.
|
||
- Merge produced the completion receipt: `merge_sha f48e363d`,
|
||
`submitted_sha 0d1585df`, `submission_ref 06G4EASR63SZ1WWRYHR62WJQS4`.
|
||
- Final state `completed`.
|
||
|
||
Freeze this as the baseline. One caveat on the reopen: Gitea returns 422 for a
|
||
review on your own pull request, and Orchestra opens the PR as the same forge
|
||
user the operator reviews as. `REQUEST_CHANGES` is therefore unreachable in this
|
||
deployment and the comment path is what was exercised. A separate bot account
|
||
for `ORCHESTRA_GITEA_TOKEN` would restore it.
|
||
|
||
Configuration this needed, for the record: `quality_gate` on `test-e2e` in both
|
||
`config.jsonc` (coordinator) and `/etc/orchestra/worker-projects.json` (worker,
|
||
which is the copy `finalize` actually reads), plus `ORCHESTRA_REVIEW_ACTORS=kami`
|
||
in the compose `.env`.
|
||
|
||
## Plan machinery, written 2026-08-28, unproven live
|
||
|
||
Four commits above `orchestra-conformance-v1`. Nothing deployed. Full detail is
|
||
in `HANDOFF-2026-08-28-plan-machinery.md`; this is the ledger entry.
|
||
|
||
| Commit | What |
|
||
|---|---|
|
||
| `822f086` | Research findings gain id and confidence. The advertised schema becomes true. |
|
||
| `57c028f` | `plan.md` replaces the four-bullet-list plan artifact. Renders verbatim into the implement launch. |
|
||
| `a221502` | Plan phase progress Orchestra establishes, plus the project verification allowlist. |
|
||
| `c76112a` | Typed plan mismatch and the Orchestra-owned reopen. |
|
||
|
||
### The burn-in it needs
|
||
|
||
Configuration first: `test-e2e` needs a `verification` block in the
|
||
coordinator's `config.jsonc`. Absent policy refuses every plan command, so
|
||
without it a plan seals and no phase can ever verify. Both halves must be
|
||
rebuilt and redeployed together, because the worker calls two new federation
|
||
routes.
|
||
|
||
Then the ladder:
|
||
|
||
```text
|
||
1. plan seals three detailed phases
|
||
-> grep the rendered launch.md for the phase-three verification command
|
||
2. phase 1 requests verification
|
||
-> worker runs the exact plan command
|
||
-> durable state says verified
|
||
3. rotation
|
||
-> successor gets the complete original plan
|
||
-> phase 1 verified, phase 2 explicitly current
|
||
4. human correction lands above the plan
|
||
5. phase 2 contradiction
|
||
-> PlanMismatchRecorded, no improvisation
|
||
-> old plan retained until the revised one seals
|
||
-> fresh launch carries the revised plan in full
|
||
6. a plan command outside project policy
|
||
-> refused at seal time, on the planner
|
||
7. a legacy plan
|
||
-> "phase progress unavailable", task still completes
|
||
```
|
||
|
||
Steps 1 and 3 are the rungs that matter for smaller local models. Steps 6 and 7
|
||
were added from what this session's implementation surfaced.
|
||
|
||
Freeze as `orchestra-plan-v1` if it passes.
|
||
|
||
### F18 is now blocking
|
||
|
||
Three more instances landed with this change: every refusal in the new plan
|
||
machinery records into the same single `last_error` slot. That is the fourth,
|
||
fifth and sixth instance. The plan burn-in will produce exactly the five-second
|
||
retry loops the bounded observation ring exists to make visible, so fix F18
|
||
before running it.
|
||
|
||
## Runs 7 to 10, the plan machinery burn-in, 2026-08-28
|
||
|
||
Both halves were rebuilt and redeployed together seven times across these runs,
|
||
ending at `38aa073`. Every revision below was verified from the running
|
||
process, not the installed file.
|
||
|
||
`test-e2e` gained a verification policy in the coordinator's `config.jsonc`:
|
||
|
||
```json
|
||
"verification": { "allowed": [["bash", "-n", "*"], ["bash", "scripts/orchestra_e2e_healthcheck.sh"]] }
|
||
```
|
||
|
||
**The policy has one configuration copy, not two.** The worker never reads it:
|
||
`requestPlanVerification` calls `PlanPhaseCommands` on the coordinator and runs
|
||
only what comes back. A `verification` field in
|
||
`/etc/orchestra/worker-projects.json` would be dead configuration that later
|
||
reads as authoritative.
|
||
|
||
### What is proven
|
||
|
||
| Rung | Evidence |
|
||
|---|---|
|
||
| Research findings are citable | Run 10: seven findings, three confidence classes, five cited by the plan |
|
||
| The plan seals as a specification | Run 8: 7940 bytes, three phases, five `run:` lines, one manual step |
|
||
| The plan renders verbatim into implement | Run 8: the whole 7940-byte document byte for byte inside a 20822-byte `launch.md`, phase 3 heading and command present |
|
||
| Orchestra establishes phase progress | Run 8: `phase-1` verified at exit `[0]`, bound to `plan_ref`, `at_sha`, `evidence_ref`, harness and lease epoch |
|
||
| A manual step holds the phase | Runs 8, 9, 10: `awaiting_manual_verification` on every phase declaring one |
|
||
| A generic comment does not satisfy it | Run 8: "looks good to me" landed as `operator_instruction`, phase 2 unmoved |
|
||
| A keyed sign-off does | Run 8: `plan_phase_verification:51afbfa9…:phase-2`, phase 2 verified |
|
||
| Rotation preserves progress | Run 8: the successor received the whole plan and the progress block |
|
||
| Verification is bound to a SHA | Run 8: three phases rendered "verified at 14654d6cab63, stale because the tree is now at 56d9b9acdd59" |
|
||
| Reverification moves with the tree | Run 10: all three phases reverified at `ec0502f` after HEAD moved |
|
||
| A plan command outside policy is refused at seal | Run 10: refused on the planner, corrected in 25 seconds, then sealed |
|
||
| A legacy plan degrades honestly | Decoded from the real CAS artifact `1e12fcf0`: zero phases, 1646 bytes of markdown, the legacy notice present, the progress block empty |
|
||
| Review, resubmission and merge still work | Run 8: PR 11, changes requested, resubmitted, merged, receipt binds `ec320c82` |
|
||
|
||
### The typed mismatch and the reopen, run 11
|
||
|
||
Proven whole, on the fourth attempt. The three failed attempts are worth
|
||
keeping, because each one was the agent being right.
|
||
|
||
The setup that worked removes the planner from the loop and the race with it.
|
||
Let research seal normally, then `POST /v1/tasks/<id>/phase` from the TUI
|
||
surface with a plan authored to assert something false about the repository.
|
||
Run 11's plan said `scripts/orchestra_e2e_healthcheck.sh` defines an
|
||
`emit_json()` helper at line 12 with three call sites and a format global. The
|
||
script has none of those.
|
||
|
||
```text
|
||
13:01:28 implement launched against the false plan
|
||
13:01:56 PlanMismatchRecorded phase-1 requested_action=replan
|
||
13:01:56 plan reopened; plan_ref still 91c4dbd4; plan_history empty
|
||
13:02:33 replanning session launched
|
||
13:03:42 replacement refused at seal, a command outside verification policy
|
||
13:04:46 corrected replacement seals
|
||
plan_ref 91c4dbd4 -> b9bcb60a
|
||
plan_history [] -> ['91c4dbd4']
|
||
plan_progress empty
|
||
13:05:28 fresh implement session, 7827 bytes of replacement plan verbatim,
|
||
phase-1 "not started"
|
||
```
|
||
|
||
The report carried an observation and no replacement plan:
|
||
|
||
```text
|
||
observed: "defines no helper at all. Line 12 is 'exit 0' inside the
|
||
--help branch. Output leaves the script from four bare
|
||
printf calls (lines 8-11 help, line 21 json, line 25 plain
|
||
success)..."
|
||
contradicts: "Phase 1 edits that helper, Phase 2 moves its three call
|
||
sites, and Phase 3 deletes the global. None of those three
|
||
targets exist in the file."
|
||
evidence: six file:line citations
|
||
requested_action: replan
|
||
```
|
||
|
||
Every authority property held. The agent reported and did not decide. A
|
||
recorded mismatch did not supersede the accepted plan, which stayed accepted
|
||
for the whole replanning window. Progress did not carry across the ref change.
|
||
|
||
`fb7135e` fired a second time inside this run, on the replacement plan, which
|
||
is the seal-time refusal working on a plan nobody set up to fail.
|
||
|
||
### Three attempts that failed, and why they were wrong
|
||
|
||
Each of these tested something that is not a plan mismatch. The agent was right
|
||
every time:
|
||
|
||
- Rewriting a file the plan depends on is *reversible*. The implementer
|
||
restored it from master and commented "Restore the byte-pinned USAGE
|
||
test_healthcheck.sh from master". The plan said the file gets no edit, an
|
||
edit appeared, and restoring it satisfied the plan.
|
||
- A human correction is *authoritative*. The brief tells the agent a decision
|
||
outranks the plan, so it followed the correction and recorded the stale plan
|
||
as an outstanding item: "The accepted plan text still specifies exit 2 and an
|
||
in-loop chain; it is stale against the correction."
|
||
|
||
- Letting a competent planner author the plan cannot produce this case at all.
|
||
It researches first, so its premises are true.
|
||
|
||
`plan-mismatch.json` is for the case where the **repository** contradicts the
|
||
plan and no human has spoken. In a repository the agent fully controls, that
|
||
has to be constructed: a false premise the implementer cannot repair and cannot
|
||
rewrite.
|
||
|
||
### Defects found, all fixed
|
||
|
||
| ID | Commit | Defect |
|
||
|---|---|---|
|
||
| F47 | `4b32080` | Delivery merges the task branch, which carries `orchestra: TASK.md`. `writeTaskFile` returned early on `os.Stat`, so every later worktree inherited the previous task's file and every release failed "TASK.md changed". Run 7 died on `retry_limit` without leaving research. |
|
||
| F48 | `49409c9` | The manual plan-phase gate had two live consumers and no producer. Every imported comment was hardcoded to `operator_instruction`, so a phase with a manual step could never be verified. A comment whose first line is `orchestra verify <phase-id>` now carries the keyed subject. |
|
||
| F49 | `f4dbcf7` | A plan whose every phase was stale printed "Every phase is verified" under three lines that each said stale. |
|
||
| F50 | `4712c7d` | The research brief never stated the finding-id format the decoder enforces. Cost one boundary. |
|
||
| F51 | `98f1b2d`, `fda78cf` | The handoff parser split on `→` only and demanded the words "tried" and "failed because". Cost four leases across run 9 on content that was exactly right. |
|
||
| F52 | `fb7135e` | A plan command outside project policy sealed anyway. The only caller of `VerificationPolicy.Allows` was `PlanPhaseCommands`, which runs when the implementer asks to verify: one phase and one rotation too late. |
|
||
| F53 | `d7e75e9` | Only a verified phase consulted `Stale`. A pending manual gate rendered "automated checks passed at 94bd45c3b5d6" against a tree at `7d04aef`. |
|
||
| F54 | `015764e` | The handoff prompt stated the 200-character limit for `NEXT`, `WHY` and `REMAINING`. The validator applies it to `OPEN Q` and `LEARNED` too. |
|
||
| F56 | `44ff35a` | `publishCaptures` iterated every session the worker held, without checking the lease. Run 10's task was blocked and unleased for twenty-six minutes while this called the coordinator every five seconds and logged "409 Conflict: lease not owned" each time. It also kept the single `last_error` slot pinned to a dead task, hiding run 11's own failures. |
|
||
| F55 | `38aa073` | **A refused handoff had no feedback loop.** `PrepareRelease` read the report, the parser refused it, the worker recorded it in health, and the next boundary read the same bytes. Run 10 spent four leases there. `answerRefusedHandoff` now says why and drops the file, gated on a typed `ErrInvalidHandoffAnswer` so transport failures keep their retry. |
|
||
|
||
F50, F51 and F54 are one shape: **a constraint the code enforces and no brief
|
||
states**. F55 is the shape that made them expensive, and it is the silent-loop
|
||
pattern `CLAUDE.md` names.
|
||
|
||
`fb7135e` has operational reach. Eight projects in `config.jsonc` carry no
|
||
`verification` block: `inference-router`, `Muzick`, `Kdrive`, `Nexus`,
|
||
`Praxis`, `Hexis`, `manga-pipeline`, `orchestra`. Their plans still seal, but
|
||
only with manual verification. A `run:` line is now refused on the planner.
|
||
|
||
### Run 11 completed the lifecycle on the final build
|
||
|
||
The same task that produced the mismatch went on through review, submission and
|
||
merge without intervention. PR 16, receipt binds merge `bf0f287a` to submitted
|
||
`5bcffb33`. That is the frozen lifecycle re-proven above every fix in this
|
||
session.
|
||
|
||
### Deployed state at the freeze
|
||
|
||
| Half | Revision |
|
||
|---|---|
|
||
| Coordinator, homesrv container | `44ff35a` |
|
||
| Worker, workpc systemd | `44ff35a` |
|
||
|
||
Run 11 itself ran on `38aa073`. `44ff35a` is one commit above it, unit-tested
|
||
and deployed, not exercised by a full run.
|
||
|
||
### Recorded, not fixed
|
||
|
||
- **The operator lifecycle actions lose a version race.** `block`, `release`
|
||
and `attention` return `task version conflict` on a leased task, every
|
||
attempt, because the handler reads the version and the worker writes between
|
||
the read and the append. They succeed once the task is quiet.
|
||
- **`master` carries a stale `TASK.md`.** F47 makes it harmless, but every
|
||
pull request now shows a cosmetic `TASK.md` diff.
|
||
- **F18 is still open**, and it cost real diagnosis time twice. The single
|
||
`last_error` slot held only the most recent failure, so run 7's three earlier
|
||
failures were overwritten before they could be read. In run 11 the slot was
|
||
pinned to a *different, blocked* task for twenty-six minutes (F56), so run
|
||
11's own expiry reason was never visible at all.
|
||
- **A release transaction whose lease has expired can never commit.** Run 10
|
||
ended this way: `PrepareRelease` succeeded, `Release` returned
|
||
`409 lease not owned`, and the transaction stayed in `anchor_pushed` across
|
||
every relaunch until `retry_limit`. The agent's completed work sat
|
||
uncommitted in the worktree throughout.
|
||
|
||
### Things that will bite
|
||
|
||
- The agent pane is on a private tmux socket:
|
||
`tmux -L orchestra capture-pane -p -t <session>:1.0`. A bare `tmux ls` does
|
||
not show it.
|
||
- A trivial task finishes three plan phases in about 105 seconds. Any
|
||
intervention timed against the implement phase will lose that race.
|
||
- `sudo` is unavailable in this sandbox, so every worker-side fix needs the
|
||
operator. Batch them: one install per run, not one per defect.
|
||
|
||
## Run 12, 2026-08-28: the expired release, proven both ways
|
||
|
||
Run 12 exists to settle one question the freeze left open: **a release
|
||
transaction whose lease has expired can never commit.** Run 10 ended that way
|
||
and lost a finished task. The fix and its guard were both proven live on
|
||
`test-e2e`, on a task built for the purpose.
|
||
|
||
### F57: an expired lease could never commit the anchor it had already pushed
|
||
|
||
`6565b9f`. A release pushes the anchor first and commits second. When the lease
|
||
died in between, the commit could never land, for two reasons at once:
|
||
|
||
- The worker sent the epoch from `w.leases`, which the expiry replay had
|
||
already deleted, so the request carried an empty epoch.
|
||
- The coordinator refused any `/handoff` without a live owned lease, and expiry
|
||
had already moved the version the worker held.
|
||
|
||
The epoch now belongs to the release transaction, so it survives the lease.
|
||
`TaskReleased` retains the ending epoch as `Task.LastLeaseEpoch`, and
|
||
`lateHandoffAccepted` lets exactly that owner commit while the task is queued,
|
||
unleased, and carrying no handoff of its own.
|
||
|
||
### The rig, and why suspending the worker cannot produce this
|
||
|
||
The ordering only exists inside one call: transaction opened, anchor pushing,
|
||
commit not yet sent. The event replay runs at the top of every tick, so any
|
||
expiry the worker learns about before pushing discards the transaction and
|
||
quarantines the session, by the F30 rule. A `SIGSTOP` therefore cannot make it
|
||
happen; the pause has to land inside the push.
|
||
|
||
What worked: poll `/var/lib/orchestra/worker-state/workpc-claude.json` at 2ms
|
||
and fire `POST /v1/tasks/<id>/release` the instant a transaction appears at
|
||
`prepared`. The forced expiry is the real expiry path, not a shortcut.
|
||
`validateTransition` accepts `reason: lease_expired` only when the payload
|
||
binds the current `harness_id` and `lease_epoch`, which is the same rule the
|
||
coordinator's own sweep obeys.
|
||
|
||
Task `06G4KENHXY12M5BNC5TXAF3MXR`, transaction `06G4KFK9EV4JZEPFEVEB65K2KM`:
|
||
|
||
```text
|
||
19:00:10.742 transaction opens, lease_epoch 06G4KF6HC66AGNG5HZZXCHRYPG, phase prepared
|
||
19:00:10.727 TaskReleased v15 reason=lease_expired epoch=06G4KF6HC66AGNG5HZZXCHRYPG surface=tui
|
||
19:00:11.662 TaskReleased v16 tx=06G4KFK9EV4JZEPFEVEB65K2KM hr=d4b203e9e5c0 surface=system
|
||
19:00:11.665 TaskLeased v17 epoch=06G4KFKD27HCR7MBS37M0T48AR hr=d4b203e9e5c0
|
||
19:00:15.524 TaskPickupValidated v18
|
||
```
|
||
|
||
The commit landed 935ms after the lease died. On `44ff35a`, v16 is
|
||
`409 lease not owned` and the anchor is stranded.
|
||
|
||
### The race guard, proven by making a successor win
|
||
|
||
Same rig, one step added: lease the task to `race-guard-probe` immediately
|
||
after the forced expiry, before the push finishes.
|
||
|
||
```text
|
||
19:01:30.539 transaction 06G4KFX15CW9S3967FV7R09GSM opens, epoch 06G4KFKD27HCR7MBS37M0T48AR
|
||
19:01:30.525 TaskReleased v22 reason=lease_expired
|
||
19:01:30.538 TaskLeased v23 harness=race-guard-probe epoch=06G4KFX15881BB6BX5X0YMV988
|
||
late commit -> 409 lease not owned, no v24 handoff, successor lease intact
|
||
```
|
||
|
||
A named probe harness is the cheap way to own a lease without starting an
|
||
agent. Nothing picks it up and it expires on the normal TTL.
|
||
|
||
### F58: a superseded release transaction retried forever
|
||
|
||
`03663f4`, and it is F57's own residue. That 409 is correct and permanent: the
|
||
late-handoff path fences on the epoch that expired, and the owner has moved on
|
||
twice. The worker kept asking every five seconds anyway, holding the pane and
|
||
pinning both `ActiveTask` and the single `last_error` slot. Run 10's task did
|
||
that for seven hours, which is also what hid run 11's failures.
|
||
|
||
`TaskLeased` now abandons a release transaction whose id the lease does not
|
||
carry, and quarantines its session. A successor pickup carries the
|
||
predecessor's own transaction id, so the recoverable predecessor F30 protects
|
||
is left alone.
|
||
|
||
### F59: a failed task kept its release transaction too
|
||
|
||
`8e37989`. F58 fires on `TaskLeased`, and a failed task is never leased again.
|
||
Run 12's own rig task proved the gap within ten minutes: two forced expiries
|
||
plus the probe's expiry pushed it to `retry_limit`, and it failed still holding
|
||
a transaction whose commit is refused permanently. `TaskFailed` now drops the
|
||
transaction and quarantines the session even when the anchor was pushed.
|
||
Blocked keeps the old rule, because a reopen still produces a successor that
|
||
can pick the anchor up.
|
||
|
||
A blocked task that is never reopened is therefore still able to loop. That is
|
||
run 10's case, and it is the one shape left that needs a hand.
|
||
|
||
### Corrected from the freeze
|
||
|
||
**The operator lifecycle actions do not lose a version race.** `block`,
|
||
`release` and `attention` are refused on a leased task by
|
||
`internal/store/store.go:885-901` when the payload omits `harness_id` and
|
||
`lease_epoch`. Ten attempts in 550ms all failed that way. Sending the two
|
||
fencing fields makes them succeed on the first try. The previous entry blamed a
|
||
read-then-append race, and that was wrong.
|
||
|
||
### Deployed state
|
||
|
||
| Half | Revision |
|
||
|---|---|
|
||
| Coordinator, homesrv container | `8e37989` |
|
||
| Worker, workpc systemd | `8e37989` |
|
||
|
||
```text
|
||
commit 8e37989526d8ea14088872138e32438b0df061c3
|
||
coordinator sha256 e2b3a2bb5ec374b7eae46586476712a16b43bee11039a910997d31f56f72903c
|
||
worker sha256 204f3c82b0b20aa2f86dfec3d117f4765678a86ce76bbfe43f9c80bfbd4cc245
|
||
```
|
||
|
||
The coordinator sha256 is `/app/orchestra` inside the container. Docker
|
||
compiles its own binary, so it never matches `build/orchestra`. F57 shipped on
|
||
`6565b9f` and F58 on `03663f4`; both were deployed and verified in turn, and
|
||
`8e37989` is the pair that is live.
|
||
|
||
The operator installed the `/etc/sudoers.d` line the freeze offered, so worker
|
||
installs no longer need a human. The batching advice still stands: each install
|
||
is still a restart of live state.
|
||
|
||
### Cleanup done, and one left
|
||
|
||
Two dead transactions were dropped from the worker state by hand, each with the
|
||
worker suspended and the file backed up:
|
||
|
||
| Task | Backup | Why the code could not clear it |
|
||
|---|---|---|
|
||
| `06G4GBSQ2WRGD5HGYPYZZ4TYH0` | `.bak-preclean-03663f4` | Run 10's. Blocked, so `TaskLeased` never comes. |
|
||
| `06G4KENHXY12M5BNC5TXAF3MXR` | `.bak-preclean-8e37989` | Run 12's rig task. Its `TaskFailed` was already behind the worker's cursor when F59 deployed. |
|
||
|
||
Both panes are still orphaned and need an operator kill. The worker no longer
|
||
holds either session, so nothing will close them:
|
||
|
||
```text
|
||
orchestra-06g4gbsq2wrgd5hgypyzz4tyh0-ec8111ad:1.0
|
||
orchestra-06g4kenhxy12m5bnc5txaf3mxr-*:1.0
|
||
tmux -L orchestra kill-session -t <session>
|
||
```
|
||
|
||
Task `06G4JX6MSQEP7N0D5JWW9EP5X4` is the other run 12 task and is `in_review` on
|
||
a pull request. It is real work and should be reviewed or failed, not cleaned.
|