diff --git a/HANDOFF-2026-08-29-truth-detector.md b/HANDOFF-2026-08-29-truth-detector.md new file mode 100644 index 0000000..5f91101 --- /dev/null +++ b/HANDOFF-2026-08-29-truth-detector.md @@ -0,0 +1,314 @@ +# Handoff: the release path is settled, and the UI became a truth detector + +Written 2026-08-29, 14:20 local (10:20 UTC). Read with `BURNIN.md` (the run +ledger, current through run 16), `DEBT-DESIGN.md`, `PLAN-SPEC-DESIGN.md`, +`AUDIT.md` and `CLAUDE.md`. The previous handoff is +`HANDOFF-2026-08-28-plan-v1.md`. + +Everything below was observed live unless it says otherwise. + +## The headline + +**The expired-release defect is fixed and proven both ways.** The whole family +around it is closed. F57 through F60 settle what happens to a release +transaction in every case. That includes the ones that used to need an operator +with a text editor. + +**Three new things exist that did not before.** A bounded observation ring on +worker health, closing F18. A read-only debt ledger projected from the event +log. An operator console rebuilt on the ethos design system. + +**The UI turned out to be a truth detector.** Nine screens were built against +real endpoints. They found four places where Orchestra has no capability to +support the intended interface. That list is the most valuable output of the +session. + +```text +orchestra-plan-v1 plan machinery proven + ↓ +orchestra-f18-baseline d6ee10f, the bounded observation ring + ↓ 6 commits +34f3c28 deployed now: release path settled, debt ledger, new UI +``` + +## Deployed state + +| Half | Revision | +|---|---| +| Coordinator, homesrv container | `34f3c28` | +| Worker, workpc systemd | `34f3c28` | + +```text +commit 34f3c2888fc7d45d190d93e1ea42501b6cd3e474 +coordinator sha256 1d32d83ec859b36e12473fab01bbfc3c769b97b4f6ae4db2841db5161e0eda19 +worker sha256 0e3877321dea8a1eeda51ccc6f3ead1a14aa5f5cae4d95b704f61c364248ec65 +``` + +`cbd6b11` is one commit above and is documentation only. Three commits are +unpushed. + +**Worker installs no longer need a human.** The operator installed the +`/etc/sudoers.d` line, so `sudo -n install …` and +`sudo -n systemctl restart orchestra-worker` both work unattended. Verify the +running revision from the journal, never the installed file. + +## The defects fixed, and how each was found + +Not one came from reading code. Every one came from a live run failing. + +| Id | Commit | What | +|---|---|---| +| F57 | `6565b9f` | An expired lease could never commit the anchor it had already pushed. The worker sent an epoch the expiry replay had deleted, and the coordinator refused any handoff without a live lease. The epoch now belongs to the transaction, `TaskReleased` retains the ending epoch, and `lateHandoffAccepted` lets exactly that owner commit while the task is queued and unleased. | +| F58 | `03663f4` | A superseded transaction retried a permanent 409 forever, holding the pane and pinning `ActiveTask`. Run 10's task did it for seven hours. `TaskLeased` now abandons a transaction whose id the lease does not carry. | +| F59 | `8e37989` | F58 fires on `TaskLeased`, and a failed task is never leased again. `TaskFailed` now drops the transaction too. | +| F60 | `3c7cf95` | The general rule the other two were reaching for. Terminal is failed or completed. Blocked keeps the transaction, because a reopen can still commit it, so `TaskBlocked` now retains the ending epoch as well. A refusal parks the commit for 30s backing off to 5 minutes, and any event about the task un-parks it. A transport failure is not an answer and retries at once. | +| F18 | `d6ee10f` | The single `last_error` slot. Worker health now carries up to sixteen distinct observations with repeat counts and first/last times, collapsing by message rather than by position. | + +### The rig that proved F57, and the guard + +```text +19:00:10.742 transaction opens at prepared, anchor pushing +19:00:10.727 TaskReleased v15 reason=lease_expired surface=tui +19:00:11.662 TaskReleased v16 the late commit, accepted 935ms after the lease died +19:00:11.665 TaskLeased v17 successor picks up the handoff +19:00:15.524 TaskPickupValidated v18 +``` + +Race guard, next boundary: force the expiry, then lease the task to a probe +harness before the push finishes. The late commit is refused, no handoff is +written, and the successor's lease stands. + +**The rig technique matters more than the rig.** Suspending the worker cannot +produce this ordering. The event replay runs at the top of every tick and +discards the transaction. The ordering exists only inside one call: +transaction opened, anchor pushing, commit not yet sent. So poll the worker +state file at 2ms and fire `POST /v1/tasks//release` the instant a +transaction appears at `prepared`. + +**A named probe harness owns a lease without starting an agent.** +`race-guard-probe` never picks anything up and expires on the normal TTL. + +## Corrections to the previous handoff + +**The operator lifecycle actions do not lose a version race.** On a leased +task, `block`, `release` and `attention` are refused by +`internal/store/store.go:885-901` when the payload omits `harness_id` and +`lease_epoch`. Ten attempts in 550ms all failed that way. Send both fencing +fields and they succeed on the first try. + +**The OpenCode Zen free tier is not blocked.** The selected model was. + +## The debt ledger + +`DEBT-DESIGN.md` answers nine design questions and carries four amendments the +operator made. Slice one is built, deployed and run. + +**Slice one writes nothing.** A read-only projection over the existing log, +plus a pure eligibility function and `GET /v1/debt`. It folded 881 events and +produced 15 candidates and 3 gaps. + +```text +v1:operational:lease_expired:workpc-opencode:lease r=41 tasks=4 +v1:operational:lease_expired:workpc-claude:lease r=29 tasks=13 +v1:operational:lease_failure:-:lease r=20 tasks=16 +v1:correctness:handoff_validation:-:lease r=5 tasks=5 +``` + +The opencode failure shape is the top item, found mechanically. Run 14 reached +the same conclusion by hand from a pane capture. + +**It reported what it cannot see, which was the point.** The 409 release loop +does not appear. That evidence lived in the F18 worker ring, and no event +carries it. Manual interventions are a non-durable gap for the same reason. + +**The first run exposed four defects in the model**, all recorded at the end of +`DEBT-DESIGN.md`: + +- the component part is too coarse for lease evidence +- `harness` is often empty on block reasons +- path normalization mangled a mismatch reference +- recurrence alone is the wrong sort order + +Layout, so the read model does not end up in the command layer: + +```text +internal/domain/debt.go types, signature, classification +internal/store/debt_projection.go the fold, and the gap report +internal/operations/debt.go CheckDebtEligibility +``` + +## The operator console + +Nine screens on the ethos system, signal violet `#8F7AE5`, routing fork motif. +Each screen was built by its own agent against a foundation with one author. +The shell, tokens and primitives could not drift into nine dialects. + +**Render before signing off.** Three bugs existed that no computed value would +have caught. All three came from looking at a screenshot: + +- The previous stylesheet fought every shared class name and leaked properties + the new rules never mention, which is how `position: fixed` survived on + `.topbar`. It is now scoped under `.legacy` and reaches only the login route. + That also stops its green accent and its `backdrop-filter` from reaching the + console. +- Go marshals a zero `time.Time` as `0001-01-01T00:00:00Z` and `omitempty` does + not omit a struct, so absent timestamps arrived populated-looking and + rendered as `739855d ago`. Stripped once in `client.ts`, with a test. +- Long machine ids overflowed their cards and painted under the next one. + +Chromium is installed at `/usr/bin/chromium`. To see a screen without a live +session, write a throwaway harness that stubs `window.fetch` and renders +`` inside a `MemoryRouter`, served by vite on a spare port. Note that +`npx` and `./node_modules/.bin/*` do not work on this filesystem: call +`node ./node_modules/vite/bin/vite.js` directly. + +## What the UI proved Orchestra cannot do + +This is the part worth acting on. Each screen refused to fake something, and +the refusals name real capability gaps. + +| Gap | Evidence | +|---|---| +| **No web-facing human-decision write path** | `Steer / Correct` is disabled. `internal/ui/ui.go`'s action switch has grant/deny approval, resubmit, handoff, release, block and complete, and nothing writes `HumanDecisionRecorded`. The spec makes steering the primary action of the task detail screen. | +| **No keystroke forwarding** | `Take control` is disabled. Only resubmit and approval grant/deny reach a live pane. | +| **Context occupancy is trapped in herdr** | Three screens independently hit it. No projection carries it. | +| **Project configuration is not served** | Repo, remote, quality gate and verification policy live only in `config.jsonc`. The projects screen can show none of it. | +| **The web cannot request a handoff for a federated task** | `RequestHandoff` needs a local coordinator and answers 503. That is the Design A guardrail working. | + +The operator's direction on these. Treat first-class direct human input as the +highest-value backend feature. Build it as `POST /v1/tasks//decisions`, +using the same durable decision semantics as Gitea comments, so Gitea, CLI and +web converge on one `HumanDecisionRecorded`. Keep take-control disabled, +because arbitrary pane input bypasses the durable authority model. Expose +occupancy through a session health projection rather than teaching the web +server about herdr. Add a read-only effective project configuration endpoint, +which F61 will also need. + +## Where the plan-machinery ladder stands + +Proven in run 14: the worker executes the **sealed plan's** commands rather +than the request's, and every `PlanPhaseVerified` binds `plan_ref`, `phase_id`, +`at_sha`, `evidence_ref`, `lease_epoch` and `harness_id`. + +Proven in run 16: a successor inherits the **whole sealed plan**, all phases +with their files, changes, verification, and the research citations. + +**Still unproven, and the next runtime item:** + +```text +mid-implement rotation +→ successor picks up in implement +→ launch context states which phases are already verified + +manual verification +→ SHA goes stale +→ re-verification + +plan mismatch +→ human decision +→ real replan, old plan retained, replacement launched +``` + +Two things make the first one hard, and both are now known: + +- **A trivial task spends about four minutes in implement**, and verifies every + phase against one tree near the end. The implementer writes the whole change + first, then verifies each phase in turn. Use a task whose implement phase + genuinely runs long. +- **A state-file edit does not survive a running worker.** It holds sessions in + memory and writes them back. Suspend, edit, then `sudo systemctl restart`. + Resuming lets the old copy win. Setting `handoff_requested` on the session is + the production rotation lever. + +Progress is withheld from a **review** successor on purpose +(`internal/agentctx/agentctx.go:321`), because an independent review must +reconstruct the change from the diff. That absence is not the defect. + +## F61 and F62, recorded and not built + +**F61: the planner learns the verification policy by refusal.** The brief says +a policy exists, not what is in it. Every plan therefore pays one refused round +trip. The recovery loop works: run 14's planner consumed the refusal and +resealed 31 seconds later. It is not a one-liner, because `agentctx.Input.Policy` is filled +from the worker's `SafeOperations` while the verification policy is +coordinator-side. The reason to promote it later is local models, which may +propose forbidden commands repeatedly because they cannot infer the allowed +substitute. + +**F62: a requested handoff nobody answers is invisible.** The rotation is +agent-driven. In run 16 the agent never wrote `HANDOFF.md`, renewals stopped, +the lease expired, and the task lost an attempt. Worker health recorded only +`agent status idle and pane unchanged`, 34 times. There is no timeout, no +retry, and no observation saying a handoff was requested and left unanswered. +The expiry is indistinguishable from an ordinary idle one. + +## Live state + +Three pull requests are open and unreviewed: + +```text +06G4M6HF1Z3EREX1X3NEKSHP24 pulls/18 +06G4M8WHGQ4P3GQMPEEH0RJRHM pulls/19 +06G4SWEVP71FYYKAV5FV0ZK5ZG pulls/20 +``` + +Both workers are online on `34f3c28` with no release transactions and no +sessions. 29 blocked `test-e2e` tasks are burn-in debris. Three `correx` tasks +are queued and unschedulable, because `correx` has no entry in the +coordinator's `config.jsonc`. + +**opencode now runs.** The model was the problem, and `hy3-free` works. It then +stops on a permission prompt, because `~/.config/opencode/opencode.jsonc` sets +`"bash": "ask"`. Orchestra can answer that exact dialog, but only when an +operator queues `grant_approval`, so unattended work stalls on the first +command. Set `"bash": "allow"` for unattended runs. That file also has no +top-level `model` key, so OpenCode picks whatever sits at the top of +`~/.local/state/opencode/model.json`, which any manual pick silently changes. + +**The opencode adapter is now a debt item, not a curiosity.** It cannot resolve +a session file. Activity therefore reads `unknown`, and the worker cannot tell +finished from never-started. The debt projection surfaced it independently as +the top recurring operational item. + +## Things that will bite + +- **Background python tasks get killed here.** Three watchers died before doing + anything. Foreground polling and the `Monitor` tool both work. +- **`/v1/events` is one line of JSON.** A `grep` for two substrings matches + across unrelated tasks. Parse it. +- **`npx` and `./node_modules/.bin/*` fail on this filesystem.** Call node + directly. +- **`rm` and `cp` are interactive.** Use `/bin/rm -f` and `install`. +- **Secrets are guarded.** Expand a token inside the container in one remote + command: + `T=$(docker exec orchestra-api printenv ORCHESTRA_TUI_TOKEN); curl -s -H "X-Orchestra-Surface: tui" -H "Authorization: Bearer $T" ...` +- **Rebuild the coordinator from a detached worktree**, and split the worktree + add, the docker build and the compose up into separate commands. +- **`deploy/build.sh` builds both halves from one commit with one stamp.** Use + it. +- **Geist is not on disk.** Both stacks fall back to system faces, and the + ethos threat model rules out the font CDN. + +## The roadmap the operator set + +```text +A. runtime correctness finish the plan-machinery live proof +B. evidence and debt durable worker observations, manual-intervention + events, then rerun the projection +C. operator surface first-class HumanDecision write API, + effective project-config read API, + session and context health projection +D. adapter fix opencode activity and session resolution +E. UI wire capabilities as backend support becomes real +``` + +The framing that ties them together is the thing to keep. The UI says what +Orchestra cannot expose or control. The debt ledger says which of those +shortcomings repeatedly costs something. The burn-in says which runtime +semantics are reliable. Those three decide what gets built next. + +One design question to settle before B's slice two writes any code. The worker +ring is bounded and lossy by construction. Ingesting it durably means deciding +whether the coordinator stores every observation as an event, or only +transitions. Storing every heartbeat's ring would write the same 41-count +observation hundreds of times.