orchestra/HANDOFF-2026-08-29-truth-detector.md

Handoff: the release path is settled, and the UI became a truth detector

Written 2026-08-29, 14:20 local (10:20 UTC). Read with BURNIN.md (the run ledger, current through run 16), DEBT-DESIGN.md, PLAN-SPEC-DESIGN.md, AUDIT.md and CLAUDE.md. The previous handoff is HANDOFF-2026-08-28-plan-v1.md.

Everything below was observed live unless it says otherwise.

The headline

The expired-release defect is fixed and proven both ways. The whole family around it is closed. F57 through F60 settle what happens to a release transaction in every case. That includes the ones that used to need an operator with a text editor.

Three new things exist that did not before. A bounded observation ring on worker health, closing F18. A read-only debt ledger projected from the event log. An operator console rebuilt on the ethos design system.

The UI turned out to be a truth detector. Nine screens were built against real endpoints. They found four places where Orchestra has no capability to support the intended interface. That list is the most valuable output of the session.

orchestra-plan-v1        plan machinery proven
        ↓
orchestra-f18-baseline   d6ee10f, the bounded observation ring
        ↓ 6 commits
34f3c28                  deployed now: release path settled, debt ledger, new UI

Deployed state

Half	Revision
Coordinator, homesrv container	34f3c28
Worker, workpc systemd	34f3c28

commit              34f3c2888fc7d45d190d93e1ea42501b6cd3e474
coordinator sha256  1d32d83ec859b36e12473fab01bbfc3c769b97b4f6ae4db2841db5161e0eda19
worker sha256       0e3877321dea8a1eeda51ccc6f3ead1a14aa5f5cae4d95b704f61c364248ec65

cbd6b11 is one commit above and is documentation only. Three commits are unpushed.

Worker installs no longer need a human. The operator installed the /etc/sudoers.d line, so sudo -n install … and sudo -n systemctl restart orchestra-worker both work unattended. Verify the running revision from the journal, never the installed file.

The defects fixed, and how each was found

Not one came from reading code. Every one came from a live run failing.

Id	Commit	What
F57	6565b9f	An expired lease could never commit the anchor it had already pushed. The worker sent an epoch the expiry replay had deleted, and the coordinator refused any handoff without a live lease. The epoch now belongs to the transaction, TaskReleased retains the ending epoch, and lateHandoffAccepted lets exactly that owner commit while the task is queued and unleased.
F58	03663f4	A superseded transaction retried a permanent 409 forever, holding the pane and pinning ActiveTask. Run 10's task did it for seven hours. TaskLeased now abandons a transaction whose id the lease does not carry.
F59	8e37989	F58 fires on TaskLeased, and a failed task is never leased again. TaskFailed now drops the transaction too.
F60	3c7cf95	The general rule the other two were reaching for. Terminal is failed or completed. Blocked keeps the transaction, because a reopen can still commit it, so TaskBlocked now retains the ending epoch as well. A refusal parks the commit for 30s backing off to 5 minutes, and any event about the task un-parks it. A transport failure is not an answer and retries at once.
F18	d6ee10f	The single last_error slot. Worker health now carries up to sixteen distinct observations with repeat counts and first/last times, collapsing by message rather than by position.

The rig that proved F57, and the guard

19:00:10.742  transaction opens at prepared, anchor pushing
19:00:10.727  TaskReleased v15  reason=lease_expired  surface=tui
19:00:11.662  TaskReleased v16  the late commit, accepted 935ms after the lease died
19:00:11.665  TaskLeased  v17   successor picks up the handoff
19:00:15.524  TaskPickupValidated v18

Race guard, next boundary: force the expiry, then lease the task to a probe harness before the push finishes. The late commit is refused, no handoff is written, and the successor's lease stands.

The rig technique matters more than the rig. Suspending the worker cannot produce this ordering. The event replay runs at the top of every tick and discards the transaction. The ordering exists only inside one call: transaction opened, anchor pushing, commit not yet sent. So poll the worker state file at 2ms and fire POST /v1/tasks/<id>/release the instant a transaction appears at prepared.

A named probe harness owns a lease without starting an agent. race-guard-probe never picks anything up and expires on the normal TTL.

Corrections to the previous handoff

The operator lifecycle actions do not lose a version race. On a leased task, block, release and attention are refused by internal/store/store.go:885-901 when the payload omits harness_id and lease_epoch. Ten attempts in 550ms all failed that way. Send both fencing fields and they succeed on the first try.

The OpenCode Zen free tier is not blocked. The selected model was.

The debt ledger

DEBT-DESIGN.md answers nine design questions and carries four amendments the operator made. Slice one is built, deployed and run.

Slice one writes nothing. A read-only projection over the existing log, plus a pure eligibility function and GET /v1/debt. It folded 881 events and produced 15 candidates and 3 gaps.

v1:operational:lease_expired:workpc-opencode:lease      r=41  tasks=4
v1:operational:lease_expired:workpc-claude:lease        r=29  tasks=13
v1:operational:lease_failure:-:lease                    r=20  tasks=16
v1:correctness:handoff_validation:-:lease               r=5   tasks=5

The opencode failure shape is the top item, found mechanically. Run 14 reached the same conclusion by hand from a pane capture.

It reported what it cannot see, which was the point. The 409 release loop does not appear. That evidence lived in the F18 worker ring, and no event carries it. Manual interventions are a non-durable gap for the same reason.

The first run exposed four defects in the model, all recorded at the end of DEBT-DESIGN.md:

• the component part is too coarse for lease evidence
• harness is often empty on block reasons
• path normalization mangled a mismatch reference
• recurrence alone is the wrong sort order

Layout, so the read model does not end up in the command layer:

internal/domain/debt.go            types, signature, classification
internal/store/debt_projection.go  the fold, and the gap report
internal/operations/debt.go        CheckDebtEligibility

The operator console

Nine screens on the ethos system, signal violet #8F7AE5, routing fork motif. Each screen was built by its own agent against a foundation with one author. The shell, tokens and primitives could not drift into nine dialects.

Render before signing off. Three bugs existed that no computed value would have caught. All three came from looking at a screenshot:

• The previous stylesheet fought every shared class name and leaked properties the new rules never mention, which is how position: fixed survived on .topbar. It is now scoped under .legacy and reaches only the login route. That also stops its green accent and its backdrop-filter from reaching the console.
• Go marshals a zero time.Time as 0001-01-01T00:00:00Z and omitempty does not omit a struct, so absent timestamps arrived populated-looking and rendered as 739855d ago. Stripped once in client.ts, with a test.
• Long machine ids overflowed their cards and painted under the next one.

Chromium is installed at /usr/bin/chromium. To see a screen without a live session, write a throwaway harness that stubs window.fetch and renders <Console> inside a MemoryRouter, served by vite on a spare port. Note that npx and ./node_modules/.bin/* do not work on this filesystem: call node ./node_modules/vite/bin/vite.js directly.

What the UI proved Orchestra cannot do

This is the part worth acting on. Each screen refused to fake something, and the refusals name real capability gaps.

Gap	Evidence
No web-facing human-decision write path	Steer / Correct is disabled. internal/ui/ui.go's action switch has grant/deny approval, resubmit, handoff, release, block and complete, and nothing writes HumanDecisionRecorded. The spec makes steering the primary action of the task detail screen.
No keystroke forwarding	Take control is disabled. Only resubmit and approval grant/deny reach a live pane.
Context occupancy is trapped in herdr	Three screens independently hit it. No projection carries it.
Project configuration is not served	Repo, remote, quality gate and verification policy live only in config.jsonc. The projects screen can show none of it.
The web cannot request a handoff for a federated task	RequestHandoff needs a local coordinator and answers 503. That is the Design A guardrail working.

The operator's direction on these. Treat first-class direct human input as the highest-value backend feature. Build it as POST /v1/tasks/<id>/decisions, using the same durable decision semantics as Gitea comments, so Gitea, CLI and web converge on one HumanDecisionRecorded. Keep take-control disabled, because arbitrary pane input bypasses the durable authority model. Expose occupancy through a session health projection rather than teaching the web server about herdr. Add a read-only effective project configuration endpoint, which F61 will also need.

Where the plan-machinery ladder stands

Proven in run 14: the worker executes the sealed plan's commands rather than the request's, and every PlanPhaseVerified binds plan_ref, phase_id, at_sha, evidence_ref, lease_epoch and harness_id.

Proven in run 16: a successor inherits the whole sealed plan, all phases with their files, changes, verification, and the research citations.

Still unproven, and the next runtime item:

mid-implement rotation
→ successor picks up in implement
→ launch context states which phases are already verified

manual verification
→ SHA goes stale
→ re-verification

plan mismatch
→ human decision
→ real replan, old plan retained, replacement launched

Two things make the first one hard, and both are now known:

• A trivial task spends about four minutes in implement, and verifies every phase against one tree near the end. The implementer writes the whole change first, then verifies each phase in turn. Use a task whose implement phase genuinely runs long.
• A state-file edit does not survive a running worker. It holds sessions in memory and writes them back. Suspend, edit, then sudo systemctl restart. Resuming lets the old copy win. Setting handoff_requested on the session is the production rotation lever.

Progress is withheld from a review successor on purpose (internal/agentctx/agentctx.go:321), because an independent review must reconstruct the change from the diff. That absence is not the defect.

F61 and F62, recorded and not built

F61: the planner learns the verification policy by refusal. The brief says a policy exists, not what is in it. Every plan therefore pays one refused round trip. The recovery loop works: run 14's planner consumed the refusal and resealed 31 seconds later. It is not a one-liner, because agentctx.Input.Policy is filled from the worker's SafeOperations while the verification policy is coordinator-side. The reason to promote it later is local models, which may propose forbidden commands repeatedly because they cannot infer the allowed substitute.

F62: a requested handoff nobody answers is invisible. The rotation is agent-driven. In run 16 the agent never wrote HANDOFF.md, renewals stopped, the lease expired, and the task lost an attempt. Worker health recorded only agent status idle and pane unchanged, 34 times. There is no timeout, no retry, and no observation saying a handoff was requested and left unanswered. The expiry is indistinguishable from an ordinary idle one.

Live state

Three pull requests are open and unreviewed:

06G4M6HF1Z3EREX1X3NEKSHP24  pulls/18
06G4M8WHGQ4P3GQMPEEH0RJRHM  pulls/19
06G4SWEVP71FYYKAV5FV0ZK5ZG  pulls/20

Both workers are online on 34f3c28 with no release transactions and no sessions. 29 blocked test-e2e tasks are burn-in debris. Three correx tasks are queued and unschedulable, because correx has no entry in the coordinator's config.jsonc.

opencode now runs. The model was the problem, and hy3-free works. It then stops on a permission prompt, because ~/.config/opencode/opencode.jsonc sets "bash": "ask". Orchestra can answer that exact dialog, but only when an operator queues grant_approval, so unattended work stalls on the first command. Set "bash": "allow" for unattended runs. That file also has no top-level model key, so OpenCode picks whatever sits at the top of ~/.local/state/opencode/model.json, which any manual pick silently changes.

The opencode adapter is now a debt item, not a curiosity. It cannot resolve a session file. Activity therefore reads unknown, and the worker cannot tell finished from never-started. The debt projection surfaced it independently as the top recurring operational item.

Things that will bite

• Background python tasks get killed here. Three watchers died before doing anything. Foreground polling and the Monitor tool both work.
• /v1/events is one line of JSON. A grep for two substrings matches across unrelated tasks. Parse it.
• npx and ./node_modules/.bin/* fail on this filesystem. Call node directly.
• rm and cp are interactive. Use /bin/rm -f and install.
• Secrets are guarded. Expand a token inside the container in one remote command: T=$(docker exec orchestra-api printenv ORCHESTRA_TUI_TOKEN); curl -s -H "X-Orchestra-Surface: tui" -H "Authorization: Bearer $T" ...
• Rebuild the coordinator from a detached worktree, and split the worktree add, the docker build and the compose up into separate commands.
• deploy/build.sh builds both halves from one commit with one stamp. Use it.
• Geist is not on disk. Both stacks fall back to system faces, and the ethos threat model rules out the font CDN.

Resume in this order

Set by the operator at the session boundary. Do not rebuild it from the commits.

1. F62 first. Make a requested-but-unanswered handoff visible and bounded. Preserve the lease while Orchestra is explicitly waiting, retry the confirmed request, then emit a causal timeout. Today it ends as generic idleness.
2. Repeat the implement-successor rung with a deliberately longer task. Get at least one PlanPhaseVerified, force the handoff while still in implement, and prove the successor sees the complete accepted plan, the verified previous phase, the first unfinished phase, and the current human authority.
3. Finish the remaining ladder. Manual verification, SHA staleness and reverification, human-decision mismatch, then a real replan with old-plan provenance.
4. Use the capability table above as backend work discovery. Do not add fake controls. Each disabled action is concrete evidence of a missing capability.
5. Continue the debt slices independently. Durable worker observations and manual-intervention events are the next evidence gaps. Not automatic maintenance yet.

What the tag means

orchestra-release-v1 -> 34f3c28
  deployed, settled lifecycle plus the truth-detector UI baseline

later HEADs
  experimental plan, debt and runtime work that must earn their own
  release proof

Use that distinction. A defect found in experimental work is not a reason to reopen settled architecture.

The roadmap the operator set

A. runtime correctness   finish the plan-machinery live proof
B. evidence and debt     durable worker observations, manual-intervention
                         events, then rerun the projection
C. operator surface      first-class HumanDecision write API,
                         effective project-config read API,
                         session and context health projection
D. adapter               fix opencode activity and session resolution
E. UI                    wire capabilities as backend support becomes real

The framing that ties them together is the thing to keep. The UI says what Orchestra cannot expose or control. The debt ledger says which of those shortcomings repeatedly costs something. The burn-in says which runtime semantics are reliable. Those three decide what gets built next.

One design question to settle before B's slice two writes any code. The worker ring is bounded and lossy by construction. Ingesting it durably means deciding whether the coordinator stores every observation as an event, or only transitions. Storing every heartbeat's ring would write the same 41-count observation hundreds of times.