kami/orchestra
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
56f5aac582cdb82d4a5a5783e13af884e4f3c46a
orchestra/AUDIT.md
T
kami 56f5aac582 Reconcile docs with reality; fix module graph, token compare, health 
Acts on the 2026-07-30 senior review (REVIEW.md findings 1, 2, 4, 5, 7).

Docs (finding 1): CLAUDE.md and AGENTS.md both claimed Design B "has zero
clients - no worker binary exists". cmd/orchestra-worker/main.go is the
deployed worker, and the non-local-herdr guardrail has landed in
Coordinator.adapterFor. Both sections rewritten; AUDIT.md gains a matching
federation-status record. The Phase 5 retention / Phase 6 deletion decision
for Design A is preserved, not flattened.

clients/ un-ignored and tracked, including the .service unit and README:
deployed code belongs in version control. Design A is NOT deleted here.

progress.md (finding 2): the file was deleted after 636ed8a, yet CLAUDE.md
instructed every session to cross-check against it. References removed from
CLAUDE.md, AGENTS.md, internal/orchestrator/rotation_test.go (comment only)
and deploy/hooks/orchestra-codex-poll.sh; AUDIT.md now carries the log role.

web/go.mod (finding 4): a module stub ends the parent package graph at the
directory boundary, so go list ./... no longer yields
web/node_modules/flatted/golang/pkg/flatted. A build tag cannot work - the
package is in the package list before tags are evaluated. Local/CI-only
breakage: Dockerfile.api builds ./cmd/orchestra by explicit path and
.dockerignore already excluded node_modules.

orchestra-worker (finding 5): untracked (8.9MB, mode 100755, still on disk);
both binaries now gitignored.

Token compare (finding 7): cmd/orchestra/main.go:139,582 use
subtle.ConstantTimeCompare, matching the authz.go idiom. The token != ""
guard stays first, so an empty configured token still means auth-disabled
rather than auth-bypass. Three further plain != secret compares remain in
internal/federation/federation.go:343,346,368 - tracked, not fixed here.

Also included from the review pass: orchestrator.go records adapter-resolution
failures in SessionHealth.LastError instead of dropping them on a bare
continue, plus an Observed flag so lease-seeded health is not mistaken for a
live reading, with a covering test. GET /v1/tasks/<id>/health now returns a
record with last_error where it previously returned a bare 404.

REVIEW.md's own second pass claimed every checkable fact held up; four did
not. AUDIT.md never contained the false Design B claim (AGENTS.md was the
second copy), the guardrail is at orchestrator.go:312 not :309, the
progress.md site list missed the codex-poll hook, and only orchestra-worker
was tracked. Verified: go build, go vet, go test, and
go list ./... | grep node_modules all clean with every change applied
together. No live herdr or pane was touched; nothing was deployed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GEugbHVYfAXFpTqDYbByEB
2026-07-30 22:51:54 +04:00

157 lines
11 KiB
Markdown
Raw Blame History

# Orchestra audit — handoff first
Audited 2026-07-30 against the working tree, spec, deployed coordinator,
workpc worker, event log, and live herdr (read-only).
**Verdict:** the source-level P0/P1/P2 fixes are in place, the live
coordinator can replay its repaired event log, and the current OpenCode worker
is connected. The system is not safe to run unattended until the controlled
QA matrix has passed for all three harnesses.
## Evidence
- `go build ./...`, `go vet ./...`, `go test ./...`, and `go test -race ./...`: pass.
- Live B17: release `seq=251`, re-lease `252`, completion `262`; the simple
probe needed six approvals, logged a `409 lease version conflict`, and
recorded `consumed:0`.
- Live follow-up: Docker owns the coordinator; the old systemd unit is
inactive. The pre-v2 repeated-`seq=1` event prefix was migrated with a
backup-preserving, explicit tool before the current coordinator replayed it.
- Deployment follow-up: coordinator and installed workpc OpenCode worker are
clean revision `d6cab133b56666f81f569f4c1c3c9a6f104088d1`. The worker service
restarted at 2026-07-30 16:07 +04, has emitted no federation failures since,
and its configured Unix-socket herdr answered `ping` with protocol `17`.
## Federation design status (updated 2026-07-30)
Design B ("workers pull tasks", `/v1/federation/*`) is the **live** design. It
is no longer clientless: `cmd/orchestra-worker/main.go` (~1,131 lines, tests in
`cmd/orchestra-worker/main_test.go`) is the deployed worker, and the workpc
OpenCode worker runs it. Any earlier statement here or in `CLAUDE.md` that
Design B "has zero clients — no worker binary exists" is obsolete.
The Design A guardrail from the 2026-07-27 decision has landed:
`Coordinator.adapterFor` (`internal/orchestrator/orchestrator.go`) refuses to
resolve an adapter for a session owned by a non-local herdr, so rotation and
cleanup can no longer validate a git anchor against the wrong machine's
checkout.
Design A itself is **retained through Phase 5** per that same decision.
`clients/herdr-bridge.go` is now tracked in git (deployed code must be in
version control); deleting Design A and `clients/` is deferred to the Phase 6
cutover.
## Remaining release blockers
- **Only OpenCode capacity is ready.** Workpc's `workpc-opencode` worker has
a configured project file and a reachable local herdr Unix socket. Homesrv
has no reachable herdr, and no Claude/Codex worker/herdr pair has been
verified, so the full three-harness matrix cannot begin yet.
- **B17 needs a fresh controlled run.** The historical probe's six approvals,
one `409 lease version conflict`, and `consumed:0` receipt came from the
old worker. They cannot be treated as evidence for the current worker until
a live OpenCode run is repeated; Claude and Codex require their own runs.
## QA handoff — next agent
1. **Preflight before creating work.** Read `GET /v1/federation/workers` and
coordinator diagnostics through an authenticated operator session. Confirm
each target worker reports revision `d6cab13`, supported `test-e2e`, and
fresh `herdr_status: reachable`; raw-ping its configured local Unix socket
with `params:{}` and confirm protocol 17. Confirm no pre-existing agents
or leased task on the target harness.
2. **OpenCode controlled continuity run.** Submit one new disposable
`test-e2e` task that makes a deterministic marker, releases at a clear turn
boundary, validates pickup from the resulting anchor, then completes.
Record event sequence, handoff ref, anchor SHA, transaction id, lease epoch,
native session evidence, quality-gate result, remote SHA, and a receipt
with known non-zero (or explicitly explained known-zero) usage. Do not use
destructive herdr calls against unrelated panes.
3. **Exercise rotation and recovery.** In separate disposable tasks, trigger
soft, hard, milestone, thrash, coordinator restart, worker restart/lost
response, stale completion, and corrupted-worker-state paths. Verify each
result is a fenced lifecycle event or durable `needs_attention`, never a
silent retry or orphaned pane. Preserve the predecessor until matching
pickup validation.
4. **Repeat on Claude and Codex only after provisioning their own reachable
worker/herdr pairs.** Do not treat OpenCode evidence as cross-harness
proof. After all runs, compare worker/coordinator revisions and checksums,
attach the artifacts/event ranges to this audit, and only then clear the
live release gate.
## P0 — correctness
| ID | Current failure | Required fix |
|---|---|---|
| H1 | **Closed 2026-07-30.** The checkout-owning worker and coordinator turn path now use `RotationStateMachine`. Workers persist harness-native identity (Claude/Codex transcript, OpenCode SQLite session id), apply soft/milestone/thrash/hard-boundary decisions, and record unknown activity/occupancy/boundary as degraded health rather than zero usage. | Verified by `go test -race ./...`; the existing turn-policy coverage now exercises the shared state machine. |
| H2 | **Closed 2026-07-30.** `PrepareRelease` verifies immutable `TASK.md`, checkpoints all repository work except protocol markers, always pushes the per-task project's scratch anchor, verifies it with `ls-remote`, and only then seals the CAS handoff. | `TestScratchCommitCapturesAllGitStatesExceptProtocolMarkers` covers staged, deleted, renamed, untracked, and protocol-marker cases; release uses the configured project remote. |
| H3 | **Closed 2026-07-30.** Worker state persists idempotent release transactions through `prepared → anchor_pushed → event_committed → pickup_validated → predecessor_retired`. Release/pickup endpoints bind transaction, anchor, and lease version; a predecessor remains mapped and is retired only after matching pickup validation. | `TestReleaseTransactionSurvivesReLeaseUntilMatchingPickup` covers transaction propagation and pickup epoch binding; full race suite passes. |
| H4 | **Closed 2026-07-30.** Every new lease carries an opaque durable `lease_epoch`; renew/release/pickup/complete validate the exact harness owner and epoch at the store boundary and federation API. Offline heartbeats retain leases until expiry, new workers require a fresh reachable local-herdr probe, and local/worker ownership loss stops or durably quarantines the old pane before its mapping is dropped. | `TestLeaseEpochFencesStaleOwnerLifecycleWrites`, `TestAvailableRequiresFreshReachableLocalHerdrHealth`, plus the full race suite cover stale re-lease/completion and health admission. |
| H5 | **Closed 2026-07-30.** `Store.Append` validates legal state/owner/epoch transitions, fsyncs the event before applying its projection, and replays projections solely from `events.jsonl` (snapshots are disposable caches). CAS, worker/federation/coordinator state use temp-file + fsync + rename; corrupt worker state aborts startup. The live legacy `/v1/harness/complete` route is retired (410); its retained compatibility handler is fenced if invoked directly. | `TestOpenRebuildsOnlyFromLogAndIgnoresCorruptSnapshot`, `TestWorkerRefusesCorruptDurableState`, and `go test -race ./...` pass. |
## P1 — autonomy and recovery
- **Recovery:** **Closed 2026-07-30.** Launch/recovery faults now emit
`TaskNeedsAttention`, retaining the durable harness owner and lease epoch.
Renew, release, expiry, and a late reconciled completion accept that same
fenced lease; worker state advances its expected aggregate version without
dropping the live session. `TaskBlocked` remains terminal for an explicit
operator block. `TestNeedsAttentionRetainsFencedLeaseForLateCompletion`
covers the durable recovery path.
- **Retries:** **Closed 2026-07-30.** Hand-off-less `TaskReleased` is the
single durable reclaim transition. It projects exponential `attempt`,
`next_retry_at`, and `failure_class`; router assignment reads those fields,
so coordinator restarts cannot reset a backoff or retry limit.
`TestReclaimPersistsAttemptAndBackoffAcrossReopen` covers replay.
- **Launch:** **Closed 2026-07-30.** Workers emit a fenced
`TaskLaunchAcknowledged` only after a local start/prompt is persisted.
Typed NACKs immediately reclaim transient unusable capacity, terminally
block invalid handoffs, and retain uncertain live panes for reconciliation.
- **Completion:** **Closed 2026-07-30.** `.orchestra/done` is explicit
intent only; the worker also requires native non-busy identity, runs its
quality gate, verifies immutable `TASK.md`, commits, pushes, and checks
the remote SHA before it emits completion.
- **Quota:** **Closed 2026-07-30.** Completion receipts contain native
per-lease deltas plus a known/unknown marker. Five-hour and weekly
projections are published from the same receipts; any bounded harness
without fresh known usage fails routing closed.
- **Approvals:** **Closed 2026-07-30.** Projects have a validated audited
`safe_operations` policy limited to worktree-local read/edit/test/Git.
Workers inject it into the task prompt; network, secrets, destructive
actions, and paths outside the worktree remain operator-gated.
- **Observability:** **Closed 2026-07-30.** Task projections now retain
lifecycle phase, last error, retry time/failure class, lease epoch, pane
state, and anchor. Release/anchor certification faults enter durable
`needs_attention` instead of disappearing through retry `continue` paths.
## P2 — performance
- **Closed 2026-07-30.** Each scheduling pass takes one atomic task/lease
snapshot, batches cached (TTL) reachability probes concurrently, and
evaluates candidate availability once. It no longer probes candidates or
scans active tasks once per queued task.
- **Closed 2026-07-30.** Active leases and per-harness, time-ordered quota
receipts are projection indexes. Availability uses indexed rolling-window
sums rather than decoding the event log; the task snapshot is a one-time
disposable compatibility cache rather than a full rewrite on every append.
- **Closed 2026-07-30.** `BenchmarkAssignPending{1K,10K}` and
`BenchmarkAppend{1K,10K}` report and enforce p95 budgets, with durable
fsync cost included in their respective paths.
## Delivery order
1. Durable event transitions + lease fencing.
2. Idempotent checkpoint/release/pickup transaction.
3. Worker-local rotation, completion, quota, and typed recovery.
4. Approval policy and performance indexes.
5. Only then: ingestion/UI expansion.
## Release gate
- **Pass:** build, vet, test, and race checks pass; unit/integration coverage
includes the defined fault and cross-machine cases.
- **Pending QA:** run the controlled soft, hard, milestone, thrash,
completion, and late-recovery paths on OpenCode, Claude, and Codex without
manual intervention for safe repository work. Follow the QA handoff above
and attach the resulting event ranges/artifacts before clearing this gate.
Reference in New Issue View Git Blame Copy Permalink