Harden lease lifecycle durability
This commit is contained in:
kami
@@ -9,8 +9,7 @@ released agent, or reject a valid completion.
|
||||
|
||||
## Evidence
|
||||
|
||||
- `go build ./...`, `go vet ./...`, `go test ./...`: pass.
|
||||
`go test -race ./...`: fails in orchestrator monitor tests.
|
||||
- `go build ./...`, `go vet ./...`, `go test ./...`, and `go test -race ./...`: pass.
|
||||
- Live B17: release `seq=251`, re-lease `252`, completion `262`; the simple
|
||||
probe needed six approvals, logged a `409 lease version conflict`, and
|
||||
recorded `consumed:0`.
|
||||
@@ -25,8 +24,8 @@ released agent, or reject a valid completion.
|
||||
| H1 | **Closed 2026-07-30.** The checkout-owning worker and coordinator turn path now use `RotationStateMachine`. Workers persist harness-native identity (Claude/Codex transcript, OpenCode SQLite session id), apply soft/milestone/thrash/hard-boundary decisions, and record unknown activity/occupancy/boundary as degraded health rather than zero usage. | Verified by `go test -race ./...`; the existing turn-policy coverage now exercises the shared state machine. |
|
||||
| H2 | **Closed 2026-07-30.** `PrepareRelease` verifies immutable `TASK.md`, checkpoints all repository work except protocol markers, always pushes the per-task project's scratch anchor, verifies it with `ls-remote`, and only then seals the CAS handoff. | `TestScratchCommitCapturesAllGitStatesExceptProtocolMarkers` covers staged, deleted, renamed, untracked, and protocol-marker cases; release uses the configured project remote. |
|
||||
| H3 | **Closed 2026-07-30.** Worker state persists idempotent release transactions through `prepared → anchor_pushed → event_committed → pickup_validated → predecessor_retired`. Release/pickup endpoints bind transaction, anchor, and lease version; a predecessor remains mapped and is retired only after matching pickup validation. | `TestReleaseTransactionSurvivesReLeaseUntilMatchingPickup` covers transaction propagation and pickup epoch binding; full race suite passes. |
|
||||
| H4 | Lease loss can create split-brain work. Worker-offline releases after heartbeat TTL, while the old worker drops release/block mappings without stopping the pane (`cmd/orchestra/main.go:384`; `cmd/orchestra-worker/main.go:592`). Worker health may say herdr is unreachable but routing checks only heartbeat. | Give every lease a durable epoch/fencing token. Accept renew/release/complete only from that owner+epoch. Reassign only after explicit relinquish or lease expiry. On ownership loss, quarantine/stop the old pane before forgetting it. Admit workers only with fresh local-herdr health. |
|
||||
| H5 | The event log can diverge from memory: `Store.Append` mutates the projection before the event write/fsync (`internal/store/store.go:328-345`). Legal lifecycle transitions are not enforced; the legacy completion endpoint is not lease-owner fenced (`cmd/orchestra/main.go:134-185`). | Validate transition+owner+epoch, append/fsync first, then project. Recover projections only from the log. Make CAS/state writes temp+fsync+rename and fail closed on corrupt worker state. Remove or fence the legacy harness endpoint. |
|
||||
| H4 | **Closed 2026-07-30.** Every new lease carries an opaque durable `lease_epoch`; renew/release/pickup/complete validate the exact harness owner and epoch at the store boundary and federation API. Offline heartbeats retain leases until expiry, new workers require a fresh reachable local-herdr probe, and local/worker ownership loss stops or durably quarantines the old pane before its mapping is dropped. | `TestLeaseEpochFencesStaleOwnerLifecycleWrites`, `TestAvailableRequiresFreshReachableLocalHerdrHealth`, plus the full race suite cover stale re-lease/completion and health admission. |
|
||||
| H5 | **Closed 2026-07-30.** `Store.Append` validates legal state/owner/epoch transitions, fsyncs the event before applying its projection, and replays projections solely from `events.jsonl` (snapshots are disposable caches). CAS, worker/federation/coordinator state use temp-file + fsync + rename; corrupt worker state aborts startup. The live legacy `/v1/harness/complete` route is retired (410); its retained compatibility handler is fenced if invoked directly. | `TestOpenRebuildsOnlyFromLogAndIgnoresCorruptSnapshot`, `TestWorkerRefusesCorruptDurableState`, and `go test -race ./...` pass. |
|
||||
|
||||
## P1 — autonomy and recovery
|
||||
|
||||
|
||||
@@ -0,0 +1,534 @@
|
||||
# Orchestra — Token-Minimal Unattended Workflow Plan
|
||||
|
||||
**Status:** proposed
|
||||
**Written:** 2026-07-29
|
||||
**Objective:** deliver the intended unattended workflow while spending model
|
||||
tokens only on useful implementation or review work. Routing, supervision,
|
||||
continuity, Git delivery, reporting, quota accounting, and operator summaries
|
||||
must be deterministic.
|
||||
|
||||
This plan is forward-looking. `AUDIT.md` remains the defect record, but claims
|
||||
in either document must be verified against the live call path and a real
|
||||
worker-owned checkout.
|
||||
|
||||
---
|
||||
|
||||
## 1. Target workflow
|
||||
|
||||
```text
|
||||
Vikunja task
|
||||
→ deterministic ingest and eligibility mapping
|
||||
→ router selects a project-compatible, reachable worker/harness
|
||||
→ worker synchronizes the correct project checkout
|
||||
→ worker writes and commits immutable TASK.md
|
||||
→ harness reads TASK.md and performs the task
|
||||
→ worker supervises lease, approvals, context, and liveness
|
||||
→ deterministic quality gate
|
||||
→ worker commits and pushes the result
|
||||
→ optional review stage
|
||||
→ transactional TaskCompleted
|
||||
→ Vikunja reflection and morning brief
|
||||
```
|
||||
|
||||
The operator is interrupted only for:
|
||||
|
||||
- a real harness approval that cannot be safely pre-authorized;
|
||||
- a blocker requiring a human decision;
|
||||
- failed deterministic gates after the configured retry policy;
|
||||
- exhausted or uncertain quota;
|
||||
- an integrity failure involving the checkout, handoff, or pushed anchor.
|
||||
|
||||
### Token boundary
|
||||
|
||||
Model tokens are permitted for:
|
||||
|
||||
- implementing the task;
|
||||
- reasoning about a blocker;
|
||||
- an explicitly requested independent review;
|
||||
- a small semantic handoff only when deterministic state is insufficient.
|
||||
|
||||
Model tokens are not permitted for:
|
||||
|
||||
- task routing or prioritization;
|
||||
- worker/project selection;
|
||||
- lease renewal;
|
||||
- Git inspection, commits, pushes, or cleanup;
|
||||
- ordinary threshold handoffs;
|
||||
- completion reports and receipts;
|
||||
- quality-gate execution;
|
||||
- quota aggregation;
|
||||
- standups, briefs, or task-source reflection.
|
||||
|
||||
---
|
||||
|
||||
## 2. Current baseline
|
||||
|
||||
As observed on 2026-07-29:
|
||||
|
||||
- Source `1ca9d64` passes `go test ./...`, `go build ./...`, and
|
||||
`go vet ./...`.
|
||||
- The coordinator reports 28 historical tasks: 22 blocked, 5 completed, and
|
||||
1 failed, with no queued or leased work.
|
||||
- One `workpc-opencode` worker is online, but its installed binary was built
|
||||
from `95a96d8` with a modified worktree, not current source.
|
||||
- The worker is fixed to `/tmp/test-e2e`; it is not safe for arbitrary
|
||||
workpc-affine projects.
|
||||
- Three tasks recorded as completed still have live idle OpenCode agents.
|
||||
- Their worker session mappings are gone, so Orchestra cannot manage those
|
||||
panes through the normal lifecycle.
|
||||
- Completed worktrees retain `.orchestra-report.md`; two contain uncommitted
|
||||
task output, and their task branches were not pushed.
|
||||
- Federated completion receipts record `consumed: 0`.
|
||||
- Worker leases expire after 30 minutes and have no renewal path.
|
||||
- Canonical handoff semantic fields are validated and stored, but successors
|
||||
are told only to inspect `TASK.md` and Git history; the semantic fields are
|
||||
not consumed.
|
||||
- Automatic Vikunja ingestion and guarded reflection are not implemented.
|
||||
|
||||
**Operational conclusion:** do not run a valuable real task until Milestones
|
||||
1–3 are proven. Do not close the existing live panes during implementation or
|
||||
investigation without explicit operator approval.
|
||||
|
||||
---
|
||||
|
||||
## 3. Design principles
|
||||
|
||||
### 3.1 One authoritative instruction
|
||||
|
||||
`TASK.md` is the only task instruction supplied to a harness. It contains:
|
||||
|
||||
- immutable task identity and source;
|
||||
- title and complete description;
|
||||
- checkable acceptance criteria;
|
||||
- project-specific quality-gate command or reference;
|
||||
- the minimal completion signal;
|
||||
- the prohibition against editing `TASK.md`.
|
||||
|
||||
The initial launch prompt should be bounded and stable:
|
||||
|
||||
> Read TASK.md at the worktree root and execute it.
|
||||
|
||||
Do not inject the title and description again. A pickup prompt may add only
|
||||
the bounded continuity facts that are not already derivable from `TASK.md`
|
||||
and the checkout.
|
||||
|
||||
### 3.2 Completion is a transaction
|
||||
|
||||
A task is not complete merely because a report marker exists. The worker
|
||||
must persist and advance these idempotent phases:
|
||||
|
||||
```text
|
||||
completion_requested
|
||||
→ gate_passed
|
||||
→ result_committed
|
||||
→ result_pushed
|
||||
→ report_uploaded
|
||||
→ completion_appended
|
||||
→ pane_closed
|
||||
→ worktree_cleaned
|
||||
```
|
||||
|
||||
Each phase records its evidence before advancing. Restart resumes the first
|
||||
unfinished phase. Repeating a completed phase must be safe.
|
||||
|
||||
`TaskCompleted` is emitted only after the pushed anchor is verified against
|
||||
the configured remote.
|
||||
|
||||
### 3.3 The plane authors mechanical evidence
|
||||
|
||||
Replace the prose `.orchestra-report.md` contract with a zero- or
|
||||
near-zero-content completion signal such as `.orchestra/done`. The worker
|
||||
generates the canonical completion report from:
|
||||
|
||||
- task ID, project, worker, harness, and pane;
|
||||
- base SHA, result SHA, branch, and remote;
|
||||
- clean/dirty state and diffstat;
|
||||
- quality-gate commands, exit codes, and timestamps;
|
||||
- artifact hashes;
|
||||
- native usage receipts for every lease interval;
|
||||
- rotation count;
|
||||
- cleanup outcome.
|
||||
|
||||
The agent does not spend a turn narrating information the worker can prove.
|
||||
|
||||
### 3.4 Leases describe ownership, not wall-clock task duration
|
||||
|
||||
Workers renew leases while they can prove all of the following:
|
||||
|
||||
- the worker is authenticated and healthy;
|
||||
- the exact task/session mapping is still durable;
|
||||
- the expected pane exists;
|
||||
- the expected harness agent remains attached;
|
||||
- the task version and lease owner still match.
|
||||
|
||||
The coordinator accepts renewal only from the current lease owner using a
|
||||
version guard. Renewal extends the deadline without creating a new attempt.
|
||||
|
||||
Expiry or worker-offline handling must:
|
||||
|
||||
1. append one observable release/failure event;
|
||||
2. route that event through the normal retry accounting path;
|
||||
3. command the owning worker to close the exact pane when reachable;
|
||||
4. retain last-pane evidence even when cleanup cannot be confirmed;
|
||||
5. prevent a replacement session from starting until ownership is resolved
|
||||
or explicitly fenced.
|
||||
|
||||
### 3.5 Workers resolve projects locally
|
||||
|
||||
A worker must not have one global repository for every task it can lease.
|
||||
Machine-local configuration resolves:
|
||||
|
||||
```text
|
||||
(machine, task.project)
|
||||
→ local base checkout
|
||||
→ local worktree root
|
||||
→ Git remote
|
||||
→ quality-gate profile
|
||||
```
|
||||
|
||||
The worker rejects a lease before starting a pane if the project is absent
|
||||
from its local registry. The router must also avoid offering such a lease.
|
||||
|
||||
### 3.6 Rotation is deterministic by default
|
||||
|
||||
Ordinary context-threshold rotation should not ask the model to write a
|
||||
handoff. The worker derives:
|
||||
|
||||
- Git anchor and branch;
|
||||
- changed files and diffstat;
|
||||
- last successful and failing commands;
|
||||
- gate results;
|
||||
- repeated tool calls and detected thrash;
|
||||
- remaining acceptance criteria from `TASK.md`.
|
||||
|
||||
A semantic handoff prompt is reserved for:
|
||||
|
||||
- thrash where the failed approach is not mechanically clear;
|
||||
- a human-facing blocker;
|
||||
- unresolved design choices;
|
||||
- manual rotation explicitly requesting judgment.
|
||||
|
||||
Milestone rotation is disabled by default in the token-minimal policy.
|
||||
Rotate for hard context pressure, thrash, quota failure, or a genuinely
|
||||
independent next phase.
|
||||
|
||||
---
|
||||
|
||||
## 4. Milestone 1 — Establish version and ownership truth
|
||||
|
||||
### Work
|
||||
|
||||
- Embed build revision, build time, and dirty status in both binaries.
|
||||
- Publish coordinator build information in diagnostics.
|
||||
- Include worker build information in registration and heartbeat.
|
||||
- Show source revision separately for coordinator and each worker.
|
||||
- Persist worker health fields: local herdr status, active task, active pane,
|
||||
last error, and check time.
|
||||
- Make worker registration declare supported projects as well as capacity.
|
||||
- Reject leases when worker project support is absent.
|
||||
- Document one deployment command and one verification command for each
|
||||
binary.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- The UI/API shows the exact coordinator and worker revisions.
|
||||
- A stale worker is visible as stale without SSH.
|
||||
- A worker configured only for `test-e2e` cannot receive a `correx` lease.
|
||||
- Restarting the coordinator preserves worker identity and pending commands.
|
||||
- `go test ./...`, `go build ./...`, and `go vet ./...` pass.
|
||||
|
||||
---
|
||||
|
||||
## 5. Milestone 2 — Fix leases and session fencing
|
||||
|
||||
### Work
|
||||
|
||||
- Add a version-guarded worker lease-renewal endpoint.
|
||||
- Renew from the worker heartbeat loop only after local pane/session
|
||||
validation.
|
||||
- Persist renewal and cleanup errors in the worker health model.
|
||||
- Route every expiry and pane-exit release through `Router.HandleEvent`.
|
||||
- Count genuine expiry/crash releases against retry policy.
|
||||
- Do not count rotations or successful stage transfers as failures.
|
||||
- Add a durable cleanup command for remote workers.
|
||||
- Retain pane identity and cleanup state after terminal task transitions.
|
||||
- Fence replacement startup when an old pane may still own the checkout.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- A disposable task can remain leased for more than 30 minutes without being
|
||||
restarted.
|
||||
- Stopping the worker causes one expiry/release and one bounded retry.
|
||||
- Restarting the worker recovers the existing session instead of starting a
|
||||
duplicate.
|
||||
- A replacement lease never overlaps an unfenced predecessor pane.
|
||||
- The task record says whether cleanup is `closed`, `unreachable`, or
|
||||
`unknown`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Milestone 3 — Transactional completion and delivery
|
||||
|
||||
### Work
|
||||
|
||||
- Introduce the minimal completion signal.
|
||||
- Define per-project gate profiles in configuration.
|
||||
- Persist the completion transaction phases.
|
||||
- Run gates in the worker-owned checkout.
|
||||
- Refuse completion on a dirty or failing result unless policy explicitly
|
||||
permits an exception.
|
||||
- Commit result work on the task branch with deterministic metadata.
|
||||
- Push the result branch and verify the remote SHA.
|
||||
- Generate the canonical completion report mechanically.
|
||||
- Upload the report and a real native usage receipt.
|
||||
- Append `TaskCompleted` only after remote verification.
|
||||
- Close the exact pane, remove transient markers, and clean the worktree.
|
||||
- Make every phase idempotent under worker or coordinator restart.
|
||||
|
||||
### Acceptance
|
||||
|
||||
A task shown as completed proves:
|
||||
|
||||
- configured gates passed;
|
||||
- the result commit exists;
|
||||
- the configured remote contains the reported SHA;
|
||||
- the completion report matches that SHA;
|
||||
- quota consumption is non-zero when the harness reported usage;
|
||||
- no managed agent remains attached;
|
||||
- the worktree is removed or has an explicit cleanup error.
|
||||
|
||||
Also prove that a failure between any two phases resumes safely without a
|
||||
duplicate prompt, duplicate commit, duplicate completion event, or lost pane.
|
||||
|
||||
---
|
||||
|
||||
## 7. Milestone 4 — Token-minimal launch and continuity
|
||||
|
||||
### Work
|
||||
|
||||
- Stop duplicating title and description in the launch prompt.
|
||||
- Extend `TASK.md` with acceptance criteria, gate profile, and completion
|
||||
signal.
|
||||
- Replace ordinary threshold handoff prompting with a mechanically generated
|
||||
checkpoint.
|
||||
- Store a compact execution ledger with the handoff artifact.
|
||||
- On pickup, inject or materialize the bounded continuity fields the
|
||||
successor actually needs.
|
||||
- Ensure `next`, `remaining`, `dead_ends`, `open_questions`, and `learned`
|
||||
are either consumed or removed from the schema; never produce unused
|
||||
context.
|
||||
- Keep semantic handoff prompts only for thrash, blockers, open decisions,
|
||||
and manual requests.
|
||||
- Add rotation policy configuration, with milestone rotation off in the
|
||||
token-minimal profile.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- The initial task description enters model context once.
|
||||
- Ordinary threshold rotation requires no extra model turn.
|
||||
- The successor receives every retained semantic handoff field.
|
||||
- Pickup verifies `TASK.md`, the pushed anchor, and the execution ledger.
|
||||
- A forced rotation continues from the recorded next action without repeating
|
||||
a recorded dead end.
|
||||
- Usage receipts distinguish productive task tokens from rotation overhead.
|
||||
|
||||
---
|
||||
|
||||
## 8. Milestone 5 — Real project-aware federation
|
||||
|
||||
### Work
|
||||
|
||||
- Define per-machine project checkout configuration.
|
||||
- Teach workers to select repo, worktree root, remote, and gate profile by
|
||||
`task.project`.
|
||||
- Validate that the selected local checkout corresponds to the configured
|
||||
project before creating a pane.
|
||||
- Account quota by host and harness.
|
||||
- Run all Git validation and cleanup on the worker that owns the checkout.
|
||||
- Retire the remaining Design A bridge deployment after confirming there are
|
||||
no consumers.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- One worker can safely execute two configured projects.
|
||||
- An unknown or mismatched project fails before model launch and consumes
|
||||
zero model tokens.
|
||||
- A real homesrv/workpc transfer validates the anchor on workpc, pushes it to
|
||||
homesrv, and picks it up from a fresh checkout.
|
||||
- Coordinator code never runs Git validation against a remote worker path.
|
||||
|
||||
---
|
||||
|
||||
## 9. Milestone 6 — Vikunja as the automatic task source
|
||||
|
||||
### Work
|
||||
|
||||
- Implement Vikunja as a provider-port adapter.
|
||||
- Map eligible lists/projects, labels, and statuses to Orchestra projects and
|
||||
capabilities.
|
||||
- Deduplicate ingest using a stable Vikunja external key.
|
||||
- Poll or subscribe without creating duplicate tasks.
|
||||
- Reflect completion, failure, and blockers back to Vikunja.
|
||||
- Mark reflected updates so they cannot loop back into ingestion.
|
||||
- Retain the Vikunja record as an external view; Orchestra remains canonical
|
||||
for execution state.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- Creating one eligible Vikunja task creates exactly one Orchestra task.
|
||||
- Repeated polling creates no duplicate.
|
||||
- Ineligible tasks consume zero model tokens.
|
||||
- Completion updates the original Vikunja task only after the pushed result
|
||||
is verified.
|
||||
- Blockers and failures are reflected with actionable evidence.
|
||||
|
||||
---
|
||||
|
||||
## 10. Milestone 7 — Optional writer/reviewer workflow
|
||||
|
||||
This stage is optional because review consumes useful model tokens by design.
|
||||
It should add no orchestration-token overhead.
|
||||
|
||||
### Default policy
|
||||
|
||||
- Mechanical tasks: deterministic gates only.
|
||||
- Normal code tasks: writer plus gates; review is opt-in by project or task.
|
||||
- High-risk/correlation tasks: writer followed by an independent reviewer.
|
||||
|
||||
### Work
|
||||
|
||||
- Represent implementation and review as explicit workflow stages rather
|
||||
than pretending the writer's first marker means terminal completion.
|
||||
- Anchor review to the pushed writer SHA.
|
||||
- Route review by capability and independence policy.
|
||||
- On acceptance, finalize the original task.
|
||||
- On requested changes, return the task to implementation with structured
|
||||
findings and the reviewed anchor.
|
||||
- Bound review loops and escalate after the configured limit.
|
||||
- Never ask a model to summarize a review already represented by structured
|
||||
findings and Git evidence.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- Reviewer always inspects the exact pushed writer SHA.
|
||||
- Writer and reviewer cannot accidentally share the same live context when
|
||||
independence is required.
|
||||
- Accepted review produces one terminal completion.
|
||||
- Requested changes preserve findings across the next writer lease.
|
||||
- Review can be disabled without changing the base task lifecycle.
|
||||
|
||||
---
|
||||
|
||||
## 11. Milestone 8 — Operator surface and morning brief
|
||||
|
||||
### Work
|
||||
|
||||
- Separate active work, needs-attention items, and historical E2E residue.
|
||||
- Lead task details with diagnosis, last activity, pane state, and next safe
|
||||
action.
|
||||
- Show completion transaction phase and pushed anchor.
|
||||
- Show worker-local herdr truth separately from coordinator probes.
|
||||
- Expose real quota windows and uncertainty.
|
||||
- Keep the morning brief a deterministic projection over events and receipts.
|
||||
- Add filters for project, source, worker, state, blocker class, and age.
|
||||
- Add a documented frontend build step that keeps embedded assets synchronized.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- An operator can distinguish queued, running, approval-blocked, retrying,
|
||||
delivery-in-progress, and completed work without reading logs.
|
||||
- Every enabled control reaches a tested backend action.
|
||||
- The brief names completed pushed SHAs, failures, blockers, pending
|
||||
approvals, quota consumption, and sync state.
|
||||
- No model call is used to generate the brief.
|
||||
|
||||
---
|
||||
|
||||
## 12. Live proof sequence
|
||||
|
||||
Use disposable tasks and avoid destructive calls against pre-existing panes.
|
||||
|
||||
1. Deploy matching coordinator and worker revisions.
|
||||
2. Confirm build revisions through the API.
|
||||
3. Register one `test-e2e` OpenCode worker.
|
||||
4. Run a task lasting longer than one lease-renewal interval.
|
||||
5. Confirm no duplicate pane or prompt.
|
||||
6. Complete it and verify gate, commit, push, report, event, pane closure, and
|
||||
worktree cleanup.
|
||||
7. Force a worker restart during the completion transaction and verify
|
||||
idempotent recovery.
|
||||
8. Force one context rotation and verify successor consumption.
|
||||
9. Force one worker-offline expiry and bounded retry.
|
||||
10. Run a task for a second configured project.
|
||||
11. Ingest one disposable Vikunja task and verify reflection.
|
||||
12. Only after all previous steps pass, queue a low-risk real task.
|
||||
|
||||
For each live proof, record:
|
||||
|
||||
- task ID and external key;
|
||||
- coordinator and worker revisions;
|
||||
- pane and harness session IDs;
|
||||
- event sequence;
|
||||
- local and remote Git SHAs;
|
||||
- gate commands and results;
|
||||
- usage receipt;
|
||||
- cleanup result.
|
||||
|
||||
---
|
||||
|
||||
## 13. Required automated coverage
|
||||
|
||||
In addition to package tests:
|
||||
|
||||
- completion transaction fault-injection tests at every phase boundary;
|
||||
- lease-renewal, expiry, fencing, and retry tests;
|
||||
- worker restart with a live session;
|
||||
- coordinator restart with an active worker;
|
||||
- project/repository mismatch rejection;
|
||||
- real handoff consumption, not validation alone;
|
||||
- non-zero federated quota receipts;
|
||||
- completion cleanup and remote-SHA verification;
|
||||
- Vikunja deduplication and reflection-loop prevention;
|
||||
- an end-to-end worker test that exercises launch through pushed completion;
|
||||
- `go test ./...`, `go build ./...`, and `go vet ./...`;
|
||||
- frontend build, typecheck, lint, and browser smoke once UI work resumes.
|
||||
|
||||
Tests that stop at an isolated package seam do not prove completion. At least
|
||||
one test must traverse the same coordinator → router → worker → herdr adapter
|
||||
→ Git → coordinator path used in deployment.
|
||||
|
||||
---
|
||||
|
||||
## 14. Deferred work
|
||||
|
||||
Do not prioritize these before the live proof sequence succeeds:
|
||||
|
||||
- UI cosmetics beyond diagnosis and safe controls;
|
||||
- additional notification channels;
|
||||
- model-generated standups or summaries;
|
||||
- automatic task decomposition;
|
||||
- cheap-model subdelegation;
|
||||
- speculative capability vocabularies;
|
||||
- schema/backend optimization that is not required by a real run;
|
||||
- more harnesses before one harness completes the lifecycle reliably.
|
||||
|
||||
---
|
||||
|
||||
## 15. Decisions to ratify
|
||||
|
||||
Defaults used by this plan:
|
||||
|
||||
1. **Token goal:** zero model-token orchestration overhead; implementation and
|
||||
explicitly requested review remain legitimate token spend.
|
||||
2. **Review:** opt-in for ordinary code, required only for configured
|
||||
high-risk work.
|
||||
3. **Completion signal:** a minimal marker; the worker authors the report.
|
||||
4. **Rotation:** no model-authored handoff for ordinary context thresholds.
|
||||
5. **Milestone rotation:** disabled in the token-minimal profile.
|
||||
6. **Affinity:** hard; a task waits rather than running on an unconfigured
|
||||
machine or checkout.
|
||||
7. **Delivery:** a verified remote SHA is mandatory before `TaskCompleted`.
|
||||
|
||||
Changing any of these defaults should update this file and the binding spec
|
||||
before implementation.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
// orchestra-password prints a bcrypt hash suitable for ORCHESTRA_WEB_PASSWORD_HASH.
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"syscall"
|
||||
|
||||
"golang.org/x/crypto/bcrypt"
|
||||
"golang.org/x/term"
|
||||
)
|
||||
|
||||
func main() {
|
||||
fmt.Fprint(os.Stderr, "Password: ")
|
||||
password, err := term.ReadPassword(int(syscall.Stdin))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil || len(password) == 0 {
|
||||
fmt.Fprintln(os.Stderr, "password is required")
|
||||
os.Exit(1)
|
||||
}
|
||||
hash, err := bcrypt.GenerateFromPassword(password, bcrypt.DefaultCost)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "hash password:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Println(string(hash))
|
||||
}
|
||||
@@ -36,6 +36,7 @@ type worker struct {
|
||||
sessions map[string]herdr.Session
|
||||
leases map[string]lease
|
||||
releases map[string]releaseTransaction
|
||||
quarantined map[string]bool
|
||||
statePath string
|
||||
hard float64
|
||||
registration federation.Worker
|
||||
@@ -79,6 +80,7 @@ func (w *worker) health(ctx context.Context) federation.WorkerHealth {
|
||||
}
|
||||
|
||||
type lease struct {
|
||||
Epoch string `json:"epoch"`
|
||||
HandoffRef string `json:"handoff_ref,omitempty"`
|
||||
TransactionID string `json:"transaction_id,omitempty"`
|
||||
AnchorSHA string `json:"anchor_sha,omitempty"`
|
||||
@@ -117,24 +119,29 @@ type completionEvidence struct {
|
||||
CompletedAt time.Time `json:"completed_at"`
|
||||
}
|
||||
type workerState struct {
|
||||
Cursor uint64 `json:"cursor"`
|
||||
Sessions map[string]herdr.Session `json:"sessions"`
|
||||
Tasks map[string]domain.Task `json:"tasks"`
|
||||
Leases map[string]lease `json:"leases"`
|
||||
Releases map[string]releaseTransaction `json:"releases"`
|
||||
Cursor uint64 `json:"cursor"`
|
||||
Sessions map[string]herdr.Session `json:"sessions"`
|
||||
Tasks map[string]domain.Task `json:"tasks"`
|
||||
Leases map[string]lease `json:"leases"`
|
||||
Releases map[string]releaseTransaction `json:"releases"`
|
||||
Quarantined map[string]bool `json:"quarantined,omitempty"`
|
||||
}
|
||||
|
||||
func (w *worker) load() {
|
||||
func (w *worker) load() error {
|
||||
b, e := os.ReadFile(w.statePath)
|
||||
if e == nil {
|
||||
var s workerState
|
||||
if json.Unmarshal(b, &s) == nil {
|
||||
w.cursor = s.Cursor
|
||||
w.sessions = s.Sessions
|
||||
w.tasks = s.Tasks
|
||||
w.leases = s.Leases
|
||||
w.releases = s.Releases
|
||||
if err := json.Unmarshal(b, &s); err != nil {
|
||||
return fmt.Errorf("corrupt worker state %s: %w", w.statePath, err)
|
||||
}
|
||||
w.cursor = s.Cursor
|
||||
w.sessions = s.Sessions
|
||||
w.tasks = s.Tasks
|
||||
w.leases = s.Leases
|
||||
w.releases = s.Releases
|
||||
w.quarantined = s.Quarantined
|
||||
} else if !errors.Is(e, os.ErrNotExist) {
|
||||
return fmt.Errorf("read worker state %s: %w", w.statePath, e)
|
||||
}
|
||||
if w.sessions == nil {
|
||||
w.sessions = map[string]herdr.Session{}
|
||||
@@ -148,16 +155,68 @@ func (w *worker) load() {
|
||||
if w.releases == nil {
|
||||
w.releases = map[string]releaseTransaction{}
|
||||
}
|
||||
if w.quarantined == nil {
|
||||
w.quarantined = map[string]bool{}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (w *worker) save() error {
|
||||
b, e := json.Marshal(workerState{Cursor: w.cursor, Sessions: w.sessions, Tasks: w.tasks, Leases: w.leases, Releases: w.releases})
|
||||
b, e := json.Marshal(workerState{Cursor: w.cursor, Sessions: w.sessions, Tasks: w.tasks, Leases: w.leases, Releases: w.releases, Quarantined: w.quarantined})
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if e := os.MkdirAll(filepath.Dir(w.statePath), 0700); e != nil {
|
||||
return e
|
||||
}
|
||||
return os.WriteFile(w.statePath, b, 0600)
|
||||
tmp := w.statePath + ".tmp"
|
||||
f, e := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0600)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if _, e = f.Write(b); e == nil {
|
||||
e = f.Sync()
|
||||
}
|
||||
if closeErr := f.Close(); e == nil {
|
||||
e = closeErr
|
||||
}
|
||||
if e != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return e
|
||||
}
|
||||
if e = os.Rename(tmp, w.statePath); e != nil {
|
||||
return e
|
||||
}
|
||||
dir, e := os.Open(filepath.Dir(w.statePath))
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
defer dir.Close()
|
||||
return dir.Sync()
|
||||
}
|
||||
|
||||
// quarantine stops a pane before its lost lease mapping can be forgotten.
|
||||
// A failed close remains durable and is retried; it is never treated as a
|
||||
// harmless cleanup error while the old harness could still be working.
|
||||
func (w *worker) quarantine(ctx context.Context, taskID string, s herdr.Session) {
|
||||
if w.herdr == nil {
|
||||
w.quarantined[taskID] = true
|
||||
return
|
||||
}
|
||||
if err := (herdr.CLIAdapter{Client: w.herdr, Harness: w.harness}).Kill(ctx, s); err != nil {
|
||||
w.quarantined[taskID] = true
|
||||
w.recordError(fmt.Errorf("quarantine %s: %w", taskID, err))
|
||||
return
|
||||
}
|
||||
delete(w.sessions, taskID)
|
||||
delete(w.quarantined, taskID)
|
||||
}
|
||||
|
||||
func (w *worker) retryQuarantines(ctx context.Context) {
|
||||
for taskID := range w.quarantined {
|
||||
if s, ok := w.sessions[taskID]; ok {
|
||||
w.quarantine(ctx, taskID, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type artifactCAS struct{ api federation.Client }
|
||||
@@ -286,6 +345,9 @@ func taskHash(t domain.Task) string { b := continuity.RenderTaskFile(t); return
|
||||
|
||||
func (w *worker) releaseReady(ctx context.Context) {
|
||||
for id, s := range w.sessions {
|
||||
if w.quarantined[id] {
|
||||
continue
|
||||
}
|
||||
if l := w.leases[id]; l.HandoffRef != "" && !l.PickupAcknowledged {
|
||||
if err := w.ackPickup(ctx, id, s); err != nil {
|
||||
w.recordError(err)
|
||||
@@ -306,7 +368,7 @@ func (w *worker) releaseReady(ctx context.Context) {
|
||||
log.Printf("upload completion %s: %v", id, err)
|
||||
continue
|
||||
}
|
||||
if err = w.api.Complete(ctx, id, ref, evidence.ResultSHA, evidence.Branch, evidence.Remote, w.leases[id].Version, w.usageReceipt(s), w.sessionEvidence(ctx, id, s)); err != nil {
|
||||
if err = w.api.Complete(ctx, id, ref, evidence.ResultSHA, evidence.Branch, evidence.Remote, w.leases[id].Epoch, w.leases[id].Version, w.usageReceipt(s), w.sessionEvidence(ctx, id, s)); err != nil {
|
||||
w.recordError(fmt.Errorf("complete %s: %w", id, err))
|
||||
log.Printf("complete %s: %v", id, err)
|
||||
continue
|
||||
@@ -439,7 +501,8 @@ func (w *worker) advanceRelease(ctx context.Context, id string, s herdr.Session)
|
||||
_ = w.save()
|
||||
}
|
||||
if tx.Phase == "anchor_pushed" {
|
||||
if err := w.api.Release(ctx, id, tx.Ref, tx.AnchorSHA, tx.ID, tx.LeaseVersion, w.sessionEvidence(ctx, id, s)); err != nil {
|
||||
l := w.leases[id]
|
||||
if err := w.api.Release(ctx, id, tx.Ref, tx.AnchorSHA, tx.ID, l.Epoch, tx.LeaseVersion, w.sessionEvidence(ctx, id, s)); err != nil {
|
||||
tx.LastError, tx.UpdatedAt = err.Error(), time.Now().UTC()
|
||||
w.releases[id] = tx
|
||||
_ = w.save()
|
||||
@@ -491,7 +554,7 @@ func (w *worker) ackPickup(ctx context.Context, id string, s herdr.Session) erro
|
||||
if l.PickupAcknowledged {
|
||||
return nil
|
||||
}
|
||||
if err := w.api.Pickup(ctx, id, l.HandoffRef, l.AnchorSHA, l.TransactionID, l.Version, w.sessionEvidence(ctx, id, s)); err != nil {
|
||||
if err := w.api.Pickup(ctx, id, l.HandoffRef, l.AnchorSHA, l.TransactionID, l.Epoch, l.Version, w.sessionEvidence(ctx, id, s)); err != nil {
|
||||
return fmt.Errorf("pickup %s acknowledgement: %w", id, err)
|
||||
}
|
||||
l.PickupAcknowledged = true
|
||||
@@ -617,7 +680,7 @@ func (w *worker) renewLeases(ctx context.Context) {
|
||||
w.recordError(fmt.Errorf("validate lease %s: %w", taskID, err))
|
||||
continue
|
||||
}
|
||||
if err := w.api.Renew(ctx, taskID, l.Version, int((30 * time.Minute).Seconds())); err != nil {
|
||||
if err := w.api.Renew(ctx, taskID, l.Epoch, l.Version, int((30 * time.Minute).Seconds())); err != nil {
|
||||
w.recordError(fmt.Errorf("renew lease %s: %w", taskID, err))
|
||||
log.Printf("renew lease %s: %v", taskID, err)
|
||||
} else {
|
||||
@@ -726,6 +789,7 @@ func (w *worker) once(ctx context.Context) error {
|
||||
if e.Type == "TaskLeased" {
|
||||
var p struct {
|
||||
HarnessID string `json:"harness_id"`
|
||||
Epoch string `json:"lease_epoch"`
|
||||
HandoffRef string `json:"handoff_ref"`
|
||||
TransactionID string `json:"transaction_id"`
|
||||
AnchorSHA string `json:"anchor_sha"`
|
||||
@@ -735,17 +799,18 @@ func (w *worker) once(ctx context.Context) error {
|
||||
UntilNS int64 `json:"until_ns"`
|
||||
}
|
||||
_ = json.Unmarshal(e.Payload, &until)
|
||||
w.leases[e.TaskID] = lease{HandoffRef: p.HandoffRef, TransactionID: p.TransactionID, AnchorSHA: p.AnchorSHA, Version: e.Version, Until: time.Unix(0, until.UntilNS)}
|
||||
w.leases[e.TaskID] = lease{Epoch: p.Epoch, HandoffRef: p.HandoffRef, TransactionID: p.TransactionID, AnchorSHA: p.AnchorSHA, Version: e.Version, Until: time.Unix(0, until.UntilNS)}
|
||||
}
|
||||
}
|
||||
if e.Type == "TaskLeaseRenewed" {
|
||||
var p struct {
|
||||
HarnessID string `json:"harness_id"`
|
||||
Epoch string `json:"lease_epoch"`
|
||||
UntilNS int64 `json:"until_ns"`
|
||||
}
|
||||
if json.Unmarshal(e.Payload, &p) == nil && p.HarnessID == w.harnessID {
|
||||
l := w.leases[e.TaskID]
|
||||
l.Version, l.Until = e.Version, time.Unix(0, p.UntilNS)
|
||||
l.Version, l.Epoch, l.Until = e.Version, p.Epoch, time.Unix(0, p.UntilNS)
|
||||
w.leases[e.TaskID] = l
|
||||
}
|
||||
}
|
||||
@@ -755,6 +820,7 @@ func (w *worker) once(ctx context.Context) error {
|
||||
if w.herdr == nil {
|
||||
delete(w.sessions, e.TaskID)
|
||||
} else if err := (herdr.CLIAdapter{Client: w.herdr, Harness: w.harness}).Kill(ctx, session); err != nil {
|
||||
w.quarantined[e.TaskID] = true
|
||||
w.recordError(fmt.Errorf("close completed pane %s: %w", e.TaskID, err))
|
||||
continue
|
||||
} else {
|
||||
@@ -796,7 +862,9 @@ func (w *worker) once(ctx context.Context) error {
|
||||
// TaskPickupValidated for its transaction. Do not erase its pane
|
||||
// mapping merely because our own release event was replayed.
|
||||
if _, releasing := w.releases[e.TaskID]; !releasing {
|
||||
delete(w.sessions, e.TaskID)
|
||||
if session, active := w.sessions[e.TaskID]; active {
|
||||
w.quarantine(ctx, e.TaskID, session)
|
||||
}
|
||||
}
|
||||
}
|
||||
if e.Seq > w.cursor {
|
||||
@@ -848,6 +916,7 @@ func (w *worker) once(ctx context.Context) error {
|
||||
w.runCommands(ctx)
|
||||
w.renewLeases(ctx)
|
||||
}
|
||||
w.retryQuarantines(ctx)
|
||||
w.releaseReady(ctx)
|
||||
if err := w.save(); err != nil {
|
||||
return err
|
||||
@@ -864,7 +933,7 @@ func (w *worker) reconcileLeases(ctx context.Context) error {
|
||||
for _, task := range tasks {
|
||||
w.tasks[task.ID] = task
|
||||
if task.State == domain.StateLeased && task.Lease != nil && task.Lease.HarnessID == w.harnessID {
|
||||
active[task.ID] = lease{HandoffRef: task.HandoffRef, TransactionID: task.ReleaseTransaction, AnchorSHA: task.ReleaseAnchor, Version: task.Version, Until: task.Lease.Until}
|
||||
active[task.ID] = lease{Epoch: task.Lease.Epoch, HandoffRef: task.HandoffRef, TransactionID: task.ReleaseTransaction, AnchorSHA: task.ReleaseAnchor, Version: task.Version, Until: task.Lease.Until}
|
||||
}
|
||||
}
|
||||
for taskID := range w.leases {
|
||||
@@ -951,11 +1020,13 @@ func main() {
|
||||
if v, err := strconv.ParseInt(os.Getenv("ORCHESTRA_CONTEXT_WINDOW"), 10, 64); err == nil && v > 0 {
|
||||
window = v
|
||||
}
|
||||
w := &worker{api: federation.Client{BaseURL: required("ORCHESTRA_URL"), WorkerID: id, Token: token, AdmitToken: os.Getenv("ORCHESTRA_FEDERATION_ADMIT_TOKEN")}, harnessID: required("ORCHESTRA_WORKER_HERDR_ID"), harness: required("ORCHESTRA_WORKER_HARNESS"), repo: repo, root: root, remote: remote, projects: projects, tasks: map[string]domain.Task{}, sessions: map[string]herdr.Session{}, leases: map[string]lease{}, releases: map[string]releaseTransaction{}, statePath: os.Getenv("ORCHESTRA_WORKER_STATE"), hard: hard, soft: soft, window: window, registration: federation.Worker{ID: id, Address: os.Getenv("ORCHESTRA_WORKER_ADDRESS"), Capacity: 1, SupportedProjects: supported, Build: buildinfo.Current()}}
|
||||
w := &worker{api: federation.Client{BaseURL: required("ORCHESTRA_URL"), WorkerID: id, Token: token, AdmitToken: os.Getenv("ORCHESTRA_FEDERATION_ADMIT_TOKEN")}, harnessID: required("ORCHESTRA_WORKER_HERDR_ID"), harness: required("ORCHESTRA_WORKER_HARNESS"), repo: repo, root: root, remote: remote, projects: projects, tasks: map[string]domain.Task{}, sessions: map[string]herdr.Session{}, leases: map[string]lease{}, releases: map[string]releaseTransaction{}, quarantined: map[string]bool{}, statePath: os.Getenv("ORCHESTRA_WORKER_STATE"), hard: hard, soft: soft, window: window, registration: federation.Worker{ID: id, Address: os.Getenv("ORCHESTRA_WORKER_ADDRESS"), Capacity: 1, SupportedProjects: supported, Build: buildinfo.Current()}}
|
||||
if w.statePath == "" {
|
||||
w.statePath = filepath.Join(w.root, ".orchestra-worker-state.json")
|
||||
}
|
||||
w.load()
|
||||
if err := w.load(); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
w.herdr = herdr.New(required("ORCHESTRA_WORKER_HERDR"))
|
||||
if id != w.harnessID {
|
||||
log.Fatal("ORCHESTRA_WORKER_ID must equal ORCHESTRA_WORKER_HERDR_ID so leases and offline recovery have one owner")
|
||||
|
||||
@@ -11,10 +11,12 @@ import (
|
||||
"orchestra/internal/domain"
|
||||
"orchestra/internal/federation"
|
||||
"orchestra/internal/herdr"
|
||||
"orchestra/internal/orchestrator"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
@@ -38,6 +40,17 @@ func TestWorkerReregistersAfterCoordinatorForgetsIt(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerRefusesCorruptDurableState(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "state.json")
|
||||
if err := os.WriteFile(path, []byte(`{"cursor":`), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w := &worker{statePath: path}
|
||||
if err := w.load(); err == nil {
|
||||
t.Fatal("corrupt worker state was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestInitialReplayDoesNotResurrectReleasedLease(t *testing.T) {
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
@@ -175,6 +188,55 @@ func TestWorkerStartsRouterIssuedLeaseInLocalGitWorktree(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestFinalizeRunsGateCommitsPushesAndVerifiesRemote(t *testing.T) {
|
||||
remote := filepath.Join(t.TempDir(), "remote.git")
|
||||
if out, err := exec.Command("git", "init", "--bare", remote).CombinedOutput(); err != nil {
|
||||
t.Fatalf("remote: %v %s", err, out)
|
||||
}
|
||||
seed := t.TempDir()
|
||||
for _, a := range [][]string{{"init", seed}, {"-C", seed, "config", "user.email", "t@t"}, {"-C", seed, "config", "user.name", "t"}, {"-C", seed, "commit", "--allow-empty", "-m", "init"}, {"-C", seed, "remote", "add", "origin", remote}, {"-C", seed, "push", "-u", "origin", "HEAD:master"}} {
|
||||
if out, err := exec.Command("git", a...).CombinedOutput(); err != nil {
|
||||
t.Fatalf("git %v: %v %s", a, err, out)
|
||||
}
|
||||
}
|
||||
repo := filepath.Join(t.TempDir(), "repo")
|
||||
if out, err := exec.Command("git", "clone", remote, repo).CombinedOutput(); err != nil {
|
||||
t.Fatalf("clone: %v %s", err, out)
|
||||
}
|
||||
for _, a := range [][]string{{"-C", repo, "config", "user.email", "t@t"}, {"-C", repo, "config", "user.name", "t"}} {
|
||||
if out, err := exec.Command("git", a...).CombinedOutput(); err != nil {
|
||||
t.Fatalf("git %v: %v %s", a, err, out)
|
||||
}
|
||||
}
|
||||
root := filepath.Join(t.TempDir(), "worktrees")
|
||||
task := domain.Task{ID: "task", Source: "s", ExternalID: "delivery", Project: "p", QualityGate: "test -f result.txt"}
|
||||
wt, err := (orchestrator.GitWorktrees{Repo: repo, Root: root}).Create(context.Background(), task)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(wt, "result.txt"), []byte("delivered"), 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Join(wt, ".orchestra"), 0755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(wt, ".orchestra", "done"), nil, 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w := &worker{harnessID: "worker", harness: "opencode", repo: repo, root: root, remote: "origin", tasks: map[string]domain.Task{task.ID: task}}
|
||||
evidence, err := w.finalize(context.Background(), task.ID, herdr.Session{PaneID: "pane", Worktree: wt, TaskFileSHA: taskHash(task)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if evidence.ResultSHA == evidence.BaseSHA || evidence.Branch != "orchestra/task" || evidence.QualityGate != task.QualityGate {
|
||||
t.Fatalf("unexpected evidence: %+v", evidence)
|
||||
}
|
||||
out, err := exec.Command("git", "-C", repo, "ls-remote", "origin", "refs/heads/orchestra/task").CombinedOutput()
|
||||
if err != nil || !strings.HasPrefix(string(out), evidence.ResultSHA+"\t") {
|
||||
t.Fatalf("remote result=%q err=%v want %s", out, err, evidence.ResultSHA)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerApprovalCommandIsRevisionBoundAndAcknowledged(t *testing.T) {
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
|
||||
+27
-33
@@ -142,12 +142,14 @@ func (h harnessCompletion) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
var p struct {
|
||||
TaskID string `json:"task_id"`
|
||||
WorkerID string `json:"worker_id"`
|
||||
LeaseEpoch string `json:"lease_epoch"`
|
||||
Harness string `json:"harness"`
|
||||
TranscriptPath string `json:"transcript_path"`
|
||||
Report string `json:"report"`
|
||||
}
|
||||
if json.NewDecoder(r.Body).Decode(&p) != nil || p.TaskID == "" || p.Report == "" || p.TranscriptPath == "" {
|
||||
http.Error(w, "task_id, transcript_path, and report are required", http.StatusBadRequest)
|
||||
if json.NewDecoder(r.Body).Decode(&p) != nil || p.TaskID == "" || p.WorkerID == "" || p.LeaseEpoch == "" || p.Report == "" || p.TranscriptPath == "" {
|
||||
http.Error(w, "task_id, worker_id, lease_epoch, transcript_path, and report are required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
t, ok := h.store.Task(p.TaskID)
|
||||
@@ -155,6 +157,10 @@ func (h harnessCompletion) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "task not found", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
if t.State != domain.StateLeased || t.Lease == nil || t.Lease.HarnessID != p.WorkerID || t.Lease.Epoch != p.LeaseEpoch {
|
||||
http.Error(w, "lease not owned", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
var usage herdr.Usage
|
||||
var err error
|
||||
switch p.Harness {
|
||||
@@ -177,7 +183,7 @@ func (h harnessCompletion) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
payload, _ := json.Marshal(map[string]any{"report_ref": ref, "receipt": map[string]any{
|
||||
payload, _ := json.Marshal(map[string]any{"report_ref": ref, "harness_id": p.WorkerID, "lease_epoch": p.LeaseEpoch, "expected_version": t.Version, "receipt": map[string]any{
|
||||
"input_tokens": usage.Input, "cache_read_tokens": usage.CacheRead,
|
||||
"cache_write_tokens": usage.CacheWrite, "output_tokens": usage.Output, "numerator": usage.Numerator(),
|
||||
}})
|
||||
@@ -381,17 +387,10 @@ func main() {
|
||||
return err
|
||||
}}.Handler())
|
||||
mux.Handle("/", webui.Handler())
|
||||
workers.OnOffline = func(w federation.Worker) {
|
||||
for _, t := range s.Tasks() {
|
||||
if t.State == domain.StateLeased && t.Lease != nil && t.Lease.HarnessID == w.ID {
|
||||
p, _ := json.Marshal(map[string]any{"reason": "worker_offline", "harness_id": w.ID})
|
||||
e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
|
||||
if err := s.Append(e); err == nil && rt != nil {
|
||||
_, _ = rt.HandleEvent(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// A missed heartbeat is not relinquishment. Releasing here used to lease
|
||||
// the same task to a successor while the old pane was still running. The
|
||||
// authoritative lease timer performs the only automatic reassignment.
|
||||
workers.OnOffline = func(w federation.Worker) { log.Printf("worker %s offline; retaining leases until expiry", w.ID) }
|
||||
go func() {
|
||||
ticker := time.NewTicker(15 * time.Second)
|
||||
defer ticker.Stop()
|
||||
@@ -560,13 +559,12 @@ func main() {
|
||||
// same session-file assumption as CLIAdapter.Occupancy — rather than
|
||||
// trusting a self-reported number.
|
||||
harnessToken := os.Getenv("ORCHESTRA_HARNESS_TOKEN")
|
||||
mux.Handle("/v1/harness/complete", harnessCompletion{store: s, token: harnessToken, route: func(e domain.Event) error {
|
||||
if rt == nil {
|
||||
return nil
|
||||
}
|
||||
_, err := rt.HandleEvent(e)
|
||||
return err
|
||||
}})
|
||||
// The unaffiliated harness hook has no durable worker identity or fencing
|
||||
// epoch, so it cannot safely mutate a leased task. Completion is accepted
|
||||
// only through the authenticated federation worker endpoint below.
|
||||
mux.HandleFunc("/v1/harness/complete", func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "legacy harness completion endpoint retired; use worker completion", http.StatusGone)
|
||||
})
|
||||
// /v1/harness/turn is the unified turn-decision endpoint (AUDIT.md Phase
|
||||
// 2 items 1-2): the Face-B stop hook posts here on every ordinary turn
|
||||
// boundary (report marker absent — /v1/harness/complete covers task
|
||||
@@ -1062,7 +1060,7 @@ func main() {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
})
|
||||
mux.HandleFunc("/v1/federation/workers/", func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost || (!strings.HasSuffix(r.URL.Path, "/heartbeat") && !strings.HasSuffix(r.URL.Path, "/renew") && !strings.HasSuffix(r.URL.Path, "/handoff") && !strings.HasSuffix(r.URL.Path, "/complete") && !strings.HasSuffix(r.URL.Path, "/captures")) {
|
||||
if r.Method != http.MethodPost || (!strings.HasSuffix(r.URL.Path, "/heartbeat") && !strings.HasSuffix(r.URL.Path, "/renew") && !strings.HasSuffix(r.URL.Path, "/handoff") && !strings.HasSuffix(r.URL.Path, "/pickup") && !strings.HasSuffix(r.URL.Path, "/complete") && !strings.HasSuffix(r.URL.Path, "/captures")) {
|
||||
http.Error(w, "not found", 404)
|
||||
return
|
||||
}
|
||||
@@ -1119,6 +1117,7 @@ func main() {
|
||||
AnchorSHA string `json:"anchor_sha"`
|
||||
TransactionID string `json:"transaction_id"`
|
||||
LeaseVersion int `json:"lease_version"`
|
||||
LeaseEpoch string `json:"lease_epoch"`
|
||||
ResultSHA string `json:"result_sha"`
|
||||
Branch string `json:"branch"`
|
||||
Remote string `json:"remote"`
|
||||
@@ -1134,7 +1133,7 @@ func main() {
|
||||
http.Error(w, "task not found", 404)
|
||||
return
|
||||
}
|
||||
ownedLease := t.State == domain.StateLeased && t.Lease != nil && t.Lease.HarnessID == parts[3]
|
||||
ownedLease := t.State == domain.StateLeased && t.Lease != nil && t.Lease.HarnessID == parts[3] && t.Lease.Epoch == b.LeaseEpoch
|
||||
// A response can be lost after the append/fsync. Retrying the exact
|
||||
// release transaction is therefore a successful no-op, never a second
|
||||
// TaskReleased event and never a reason to discard the predecessor.
|
||||
@@ -1142,12 +1141,7 @@ func main() {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
return
|
||||
}
|
||||
// A prompt timeout can block the coordinator after herdr already
|
||||
// accepted the request. If that same authenticated worker later reports
|
||||
// a durable completion, reconcile it rather than preserving a known
|
||||
// false blocked state. No other blocked task is admitted here.
|
||||
recoverableBlocked := strings.HasSuffix(r.URL.Path, "/complete") && t.State == domain.StateBlocked && t.LastHarness == parts[3]
|
||||
if !ownedLease && !recoverableBlocked {
|
||||
if !ownedLease {
|
||||
http.Error(w, "lease not owned", 409)
|
||||
return
|
||||
}
|
||||
@@ -1160,7 +1154,7 @@ func main() {
|
||||
if ttl == 0 {
|
||||
ttl = int((30 * time.Minute).Seconds())
|
||||
}
|
||||
e, err := s.RenewLease(b.TaskID, parts[3], b.ExpectedVersion, time.Duration(ttl)*time.Second)
|
||||
e, err := s.RenewLease(b.TaskID, parts[3], b.LeaseEpoch, b.ExpectedVersion, time.Duration(ttl)*time.Second)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusConflict)
|
||||
return
|
||||
@@ -1181,7 +1175,7 @@ func main() {
|
||||
http.Error(w, "lease version conflict", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
p, _ := json.Marshal(map[string]any{"transaction_id": b.TransactionID, "handoff_ref": b.HandoffRef, "anchor_sha": b.AnchorSHA, "harness_id": parts[3], "lease_version": b.LeaseVersion, "expected_version": t.Version, "session_evidence": b.SessionEvidence})
|
||||
p, _ := json.Marshal(map[string]any{"transaction_id": b.TransactionID, "handoff_ref": b.HandoffRef, "anchor_sha": b.AnchorSHA, "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "lease_version": b.LeaseVersion, "expected_version": t.Version, "session_evidence": b.SessionEvidence})
|
||||
e := domain.Event{ID: id(), Type: "TaskPickupValidated", TaskID: b.TaskID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
|
||||
if err := s.Append(e); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusConflict)
|
||||
@@ -1206,7 +1200,7 @@ func main() {
|
||||
if _, ok := b.Receipt["consumed"]; !ok {
|
||||
b.Receipt["consumed"] = 0
|
||||
}
|
||||
p, _ := json.Marshal(map[string]any{"report_ref": b.HandoffRef, "receipt": b.Receipt, "result_sha": b.ResultSHA, "branch": b.Branch, "remote": b.Remote, "session_evidence": b.SessionEvidence})
|
||||
p, _ := json.Marshal(map[string]any{"report_ref": b.HandoffRef, "receipt": b.Receipt, "result_sha": b.ResultSHA, "branch": b.Branch, "remote": b.Remote, "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "expected_version": t.Version, "session_evidence": b.SessionEvidence})
|
||||
e := domain.Event{ID: id(), Type: "TaskCompleted", TaskID: b.TaskID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
|
||||
if err := s.Append(e); err != nil {
|
||||
http.Error(w, err.Error(), 409)
|
||||
@@ -1233,7 +1227,7 @@ func main() {
|
||||
http.Error(w, "release transaction and current lease version required", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
p, _ := json.Marshal(map[string]any{"handoff_ref": b.HandoffRef, "harness_id": parts[3], "anchor_sha": b.AnchorSHA, "transaction_id": b.TransactionID, "expected_version": t.Version, "session_evidence": b.SessionEvidence})
|
||||
p, _ := json.Marshal(map[string]any{"handoff_ref": b.HandoffRef, "harness_id": parts[3], "lease_epoch": b.LeaseEpoch, "anchor_sha": b.AnchorSHA, "transaction_id": b.TransactionID, "expected_version": t.Version, "session_evidence": b.SessionEvidence})
|
||||
e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: b.TaskID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
|
||||
if err := s.Append(e); err != nil {
|
||||
http.Error(w, err.Error(), 409)
|
||||
|
||||
@@ -41,6 +41,20 @@ func TestFederatedReachabilityDefersRemoteHerdrToWorkerHeartbeat(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoordinatorOwnsOnlyLocalHerdrInFederationMode(t *testing.T) {
|
||||
local := registry.Herdr{ID: "homesrv-opencode", MachineID: "homesrv"}
|
||||
remote := registry.Herdr{ID: "workpc-opencode", MachineID: "workpc"}
|
||||
if !coordinatorOwnsHerdr(local, "homesrv") {
|
||||
t.Fatal("coordinator does not own its local herdr")
|
||||
}
|
||||
if coordinatorOwnsHerdr(remote, "homesrv") {
|
||||
t.Fatal("coordinator claimed a worker-owned remote herdr")
|
||||
}
|
||||
if !coordinatorOwnsHerdr(remote, "") {
|
||||
t.Fatal("single-machine mode should retain legacy local ownership")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHarnessCompletionBuildsReceiptAndQuotaFromTranscript(t *testing.T) {
|
||||
s, err := store.Open(t.TempDir())
|
||||
if err != nil {
|
||||
@@ -56,7 +70,8 @@ func TestHarnessCompletionBuildsReceiptAndQuotaFromTranscript(t *testing.T) {
|
||||
if err := os.WriteFile(transcript, []byte(`{"message":{"usage":{"input_tokens":100,"cache_read_input_tokens":20,"cache_creation_input_tokens":5,"output_tokens":7}}}`+"\n"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"task_id": "done", "transcript_path": transcript, "report": "# done"})
|
||||
task, _ := s.Task("done")
|
||||
body, _ := json.Marshal(map[string]string{"task_id": "done", "worker_id": "local-claude", "lease_epoch": task.Lease.Epoch, "transcript_path": transcript, "report": "# done"})
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/harness/complete", bytes.NewReader(body))
|
||||
res := httptest.NewRecorder()
|
||||
harnessCompletion{store: s, token: "secret"}.ServeHTTP(res, req)
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# Deployment verification
|
||||
|
||||
Both binaries embed their Git revision, UTC build time, and dirty flag. Build
|
||||
the coordinator with `deploy/redeploy.sh`; it installs and restarts the local
|
||||
`orchestra.service`.
|
||||
|
||||
## Browser operator login
|
||||
|
||||
The browser UI requires `ORCHESTRA_WEB_USERNAME` and
|
||||
`ORCHESTRA_WEB_PASSWORD_HASH`. Generate a bcrypt hash without putting the
|
||||
password in shell history:
|
||||
|
||||
```sh
|
||||
go run ./cmd/orchestra-password
|
||||
```
|
||||
|
||||
Set the emitted hash in the service environment along with the chosen
|
||||
username, then restart the coordinator. `ORCHESTRA_WEB_TOKEN` is not used by
|
||||
the browser UI anymore.
|
||||
|
||||
Build a worker for staging on workpc with:
|
||||
|
||||
```sh
|
||||
revision=$(git rev-parse HEAD)
|
||||
build_time=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
dirty=false; test -z "$(git status --porcelain)" || dirty=true
|
||||
go build -ldflags "-X orchestra/internal/buildinfo.Revision=$revision -X orchestra/internal/buildinfo.Time=$build_time -X orchestra/internal/buildinfo.Dirty=$dirty" -o orchestra-worker ./cmd/orchestra-worker
|
||||
scp orchestra-worker workpc:~/orchestra-deploy/orchestra-worker
|
||||
ssh workpc 'sha256sum ~/orchestra-deploy/orchestra-worker'
|
||||
```
|
||||
|
||||
The worker receives only the path to a normal project configuration file:
|
||||
`ORCHESTRA_WORKER_PROJECT_CONFIG_FILE=/etc/orchestra/worker-projects.json`.
|
||||
That file contains a JSON object whose project entries contain `repo`,
|
||||
`worktree_root`, and `remote`; mount or provision it like any other worker
|
||||
configuration. The legacy single-checkout `ORCHESTRA_WORKER_PROJECTS` comma
|
||||
list remains supported for one existing checkout. An absent project is
|
||||
ineligible for routing.
|
||||
|
||||
Verify the coordinator at `GET /v1/admin/diagnostics` with the normal admin
|
||||
credential: its `build` object is the coordinator provenance. `GET
|
||||
/v1/federation/workers` shows every worker's `build`, supported projects, and
|
||||
worker-local health without SSH.
|
||||
@@ -10,8 +10,9 @@
|
||||
// Required — a project with no affinity can't be routed.
|
||||
"repo": "/var/lib/orchestra/repos/correx.git", // Optional per-project git repo path.
|
||||
// Overrides the global ORCHESTRA_REPO default.
|
||||
"worktree_root": "/var/lib/orchestra/worktrees/correx" // Optional per-project worktree dir.
|
||||
"worktree_root": "/var/lib/orchestra/worktrees/correx", // Optional per-project worktree dir.
|
||||
// Overrides global ORCHESTRA_WORKTREE_ROOT.
|
||||
"quality_gate": "go test ./... && go vet ./..." // Worker runs this before deterministic delivery.
|
||||
},
|
||||
{
|
||||
"id": "maven",
|
||||
|
||||
+5
-1
@@ -7,7 +7,11 @@ trap 'rm -f -- "$tmp_bin"' EXIT
|
||||
|
||||
cd "$repo_dir"
|
||||
echo "Building Orchestra..."
|
||||
go build -o "$tmp_bin" ./cmd/orchestra
|
||||
revision="$(git rev-parse HEAD)"
|
||||
build_time="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
dirty=false
|
||||
if [[ -n "$(git status --porcelain)" ]]; then dirty=true; fi
|
||||
go build -ldflags "-X orchestra/internal/buildinfo.Revision=$revision -X orchestra/internal/buildinfo.Time=$build_time -X orchestra/internal/buildinfo.Dirty=$dirty" -o "$tmp_bin" ./cmd/orchestra
|
||||
|
||||
echo "Installing /usr/local/bin/orchestra..."
|
||||
sudo install -o root -g root -m 0755 "$tmp_bin" /usr/local/bin/orchestra
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"test-e2e": {
|
||||
"repo": "/srv/orchestra/repos/test-e2e",
|
||||
"worktree_root": "/srv/orchestra/worktrees/test-e2e",
|
||||
"remote": "origin",
|
||||
"quality_gate": "go test ./..."
|
||||
},
|
||||
"correx": {
|
||||
"repo": "/srv/orchestra/repos/correx",
|
||||
"worktree_root": "/srv/orchestra/worktrees/correx",
|
||||
"remote": "origin",
|
||||
"quality_gate": "go test ./... && go vet ./..."
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,10 @@
|
||||
module orchestra
|
||||
|
||||
go 1.22
|
||||
|
||||
require (
|
||||
golang.org/x/crypto v0.29.0
|
||||
golang.org/x/term v0.26.0
|
||||
)
|
||||
|
||||
require golang.org/x/sys v0.27.0 // indirect
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
golang.org/x/crypto v0.29.0 h1:L5SG1JTTXupVV3n6sUqMTeWbjAyfPwoda2DLX8J8FrQ=
|
||||
golang.org/x/crypto v0.29.0/go.mod h1:+F4F4N5hv6v38hfeYwTdx20oUvLLc+QfrE9Ax9HtgRg=
|
||||
golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U=
|
||||
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
|
||||
golang.org/x/sys v0.27.0 h1:wBqf8DvsY9Y/2P8gAfPDEYNuS30J4lPHJxXSb/nJZ+s=
|
||||
golang.org/x/sys v0.27.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
|
||||
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||
golang.org/x/term v0.26.0 h1:WEQa6V3Gja/BhNxg540hBip/kkaYtRg3cxg4oXSw4AU=
|
||||
golang.org/x/term v0.26.0/go.mod h1:Si5m1o57C5nBNQo5z1iq+XDijt21BDBDp2bK0QI8e3E=
|
||||
golang.org/x/term v0.27.0 h1:WP60Sv1nlK1T6SupCHbXzSaN0b9wUmsPoRS9b61A23Q=
|
||||
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"orchestra/internal/authz"
|
||||
"orchestra/internal/buildinfo"
|
||||
"orchestra/internal/domain"
|
||||
"orchestra/internal/provider"
|
||||
"orchestra/internal/store"
|
||||
@@ -51,6 +52,7 @@ type ProbeFunc func() (bool, string)
|
||||
type Server struct {
|
||||
Store *store.Store
|
||||
RouterReady bool
|
||||
Build buildinfo.Info
|
||||
Probes map[string]ProbeFunc
|
||||
Providers map[string]*provider.Supervisor
|
||||
}
|
||||
@@ -95,7 +97,7 @@ func (s *Server) Diagnostics(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
tasks := s.Store.Tasks()
|
||||
events := s.Store.Events(0)
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"tasks": len(tasks), "events": len(events), "last_seq": func() uint64 {
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"build": s.Build, "tasks": len(tasks), "events": len(events), "last_seq": func() uint64 {
|
||||
if len(events) == 0 {
|
||||
return 0
|
||||
}
|
||||
|
||||
+51
-25
@@ -12,6 +12,8 @@ import (
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"golang.org/x/crypto/bcrypt"
|
||||
)
|
||||
|
||||
type Surface string
|
||||
@@ -72,18 +74,44 @@ func AuthorizeEvent(s Surface, typ string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// SessionCookie carries a browser's proof of the Web-surface token. A
|
||||
// top-level document load cannot set an Authorization header, so a
|
||||
// bearer-only gate forces operators to run the UI with no token at all
|
||||
// (AUDIT.md B18). The cookie is the browser-presentable equivalent; it is
|
||||
// never a second credential, only a receipt for the same token.
|
||||
// SessionCookie carries a browser's proof of a successful Web login. A
|
||||
// top-level document load cannot set an Authorization header, so browser
|
||||
// credentials are exchanged once for this HttpOnly receipt.
|
||||
const SessionCookie = "orchestra_session"
|
||||
|
||||
// SessionPath is the one Web-surface endpoint exempt from the token gate,
|
||||
// because it *is* the token check: it verifies the Web token itself and
|
||||
// exchanges it for a cookie. Gating it would make login unreachable.
|
||||
// SessionPath is the one Web-surface endpoint exempt from the session gate,
|
||||
// because it verifies login credentials and exchanges them for a cookie.
|
||||
const SessionPath = "/v1/ui/session"
|
||||
|
||||
// WebCredentials is the single configured browser operator identity. Only a
|
||||
// bcrypt password hash is accepted; Orchestra has no self-service account
|
||||
// creation or password-reset surface.
|
||||
type WebCredentials struct {
|
||||
Username string
|
||||
PasswordHash string
|
||||
}
|
||||
|
||||
func (c WebCredentials) Validate() error {
|
||||
if strings.TrimSpace(c.Username) == "" {
|
||||
return fmt.Errorf("web username is required")
|
||||
}
|
||||
if c.PasswordHash == "" {
|
||||
return fmt.Errorf("web password hash is required")
|
||||
}
|
||||
if _, err := bcrypt.Cost([]byte(c.PasswordHash)); err != nil {
|
||||
return fmt.Errorf("web password hash must be bcrypt: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Authenticate always performs bcrypt, even for an unknown username, so the
|
||||
// response does not reveal whether the configured username was correct.
|
||||
func (c WebCredentials) Authenticate(username, password string) bool {
|
||||
passwordOK := bcrypt.CompareHashAndPassword([]byte(c.PasswordHash), []byte(password)) == nil
|
||||
usernameOK := subtle.ConstantTimeCompare([]byte(username), []byte(c.Username)) == 1
|
||||
return passwordOK && usernameOK
|
||||
}
|
||||
|
||||
// Sessions issues and validates those receipts. Values are random and stored
|
||||
// hashed, so a leaked snapshot of this map does not yield a usable cookie.
|
||||
type Sessions struct {
|
||||
@@ -161,9 +189,9 @@ func HTTP(tokens map[Surface]string, next http.Handler) http.Handler {
|
||||
return HTTPWithSessions(tokens, nil, next)
|
||||
}
|
||||
|
||||
// HTTPWithSessions additionally accepts a valid session cookie in place of a
|
||||
// Bearer token, but only for the Web surface — every non-browser surface
|
||||
// still has to present the token directly.
|
||||
// HTTPWithSessions accepts a valid browser session cookie only for the Web
|
||||
// surface. Supplying Sessions makes Web authentication mandatory even when
|
||||
// the legacy Web bearer-token slot is empty.
|
||||
func HTTPWithSessions(tokens map[Surface]string, sessions *Sessions, next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
// Federation has per-worker credentials, not one shared surface token.
|
||||
@@ -195,26 +223,24 @@ func HTTPWithSessions(tokens map[Surface]string, sessions *Sessions, next http.H
|
||||
if s == System {
|
||||
s = Web
|
||||
}
|
||||
if expected := tokens[s]; expected != "" && subtle.ConstantTimeCompare([]byte(r.Header.Get("Authorization")), []byte("Bearer "+expected)) != 1 {
|
||||
if s == Web && sessions != nil {
|
||||
ok := false
|
||||
if s == Web && sessions != nil {
|
||||
if c, err := r.Cookie(SessionCookie); err == nil {
|
||||
ok = sessions.Valid(c.Value)
|
||||
}
|
||||
// The login endpoint authenticates itself, and the SPA shell
|
||||
// must load before a browser can present anything. Static
|
||||
// assets are not secrets; every /v1/ control path stays gated.
|
||||
if r.URL.Path == SessionPath {
|
||||
ok = true
|
||||
}
|
||||
if !strings.HasPrefix(r.URL.Path, "/v1/") && (r.Method == http.MethodGet || r.Method == http.MethodHead) {
|
||||
ok = true
|
||||
}
|
||||
if c, err := r.Cookie(SessionCookie); err == nil {
|
||||
ok = sessions.Valid(c.Value)
|
||||
}
|
||||
// The login endpoint authenticates itself, and the SPA shell must
|
||||
// load before a browser can present a session. Static assets are not
|
||||
// secrets; every other /v1/ control path remains session-gated.
|
||||
if r.URL.Path == SessionPath || (!strings.HasPrefix(r.URL.Path, "/v1/") && (r.Method == http.MethodGet || r.Method == http.MethodHead)) {
|
||||
ok = true
|
||||
}
|
||||
if !ok {
|
||||
http.Error(w, "unauthorized surface", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
} else if expected := tokens[s]; expected != "" && subtle.ConstantTimeCompare([]byte(r.Header.Get("Authorization")), []byte("Bearer "+expected)) != 1 {
|
||||
http.Error(w, "unauthorized surface", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
if (s == Telegram || s == Ntfy) && r.Method != http.MethodGet && r.Method != http.MethodHead {
|
||||
http.Error(w, "notify-only surface", http.StatusForbidden)
|
||||
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"golang.org/x/crypto/bcrypt"
|
||||
)
|
||||
|
||||
func TestSurfaceCapabilities(t *testing.T) {
|
||||
@@ -51,7 +53,9 @@ func TestSystemSurfaceDowngradedByHTTPMiddleware(t *testing.T) {
|
||||
// B18: the web UI is a full control plane. A session cookie must be an
|
||||
// alternative *presentation* of the Web token, never a widening of it.
|
||||
func TestWebSessionCookieGatesControlPathsOnly(t *testing.T) {
|
||||
tokens := map[Surface]string{Web: "secret"}
|
||||
// An empty legacy Web bearer-token slot must not open the browser surface:
|
||||
// providing Sessions means the caller needs a session cookie.
|
||||
tokens := map[Surface]string{}
|
||||
sessions := &Sessions{}
|
||||
h := HTTPWithSessions(tokens, sessions, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
@@ -96,6 +100,28 @@ func TestWebSessionCookieGatesControlPathsOnly(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestWebCredentialsAuthenticate(t *testing.T) {
|
||||
hash, err := bcrypt.GenerateFromPassword([]byte("correct horse battery staple"), bcrypt.MinCost)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c := WebCredentials{Username: "operator", PasswordHash: string(hash)}
|
||||
if err := c.Validate(); err != nil {
|
||||
t.Fatalf("Validate: %v", err)
|
||||
}
|
||||
if !c.Authenticate("operator", "correct horse battery staple") {
|
||||
t.Fatal("correct credentials rejected")
|
||||
}
|
||||
for _, attempt := range []struct{ username, password string }{{"operator", "wrong"}, {"other", "correct horse battery staple"}} {
|
||||
if c.Authenticate(attempt.username, attempt.password) {
|
||||
t.Fatalf("invalid credentials accepted: %+v", attempt)
|
||||
}
|
||||
}
|
||||
if err := (WebCredentials{Username: "operator", PasswordHash: "not-a-bcrypt-hash"}).Validate(); err == nil {
|
||||
t.Fatal("invalid bcrypt hash accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFederationRequestsUseTheirOwnCredentials(t *testing.T) {
|
||||
tokens := map[Surface]string{Web: "web-secret"}
|
||||
h := HTTPWithSessions(tokens, nil, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
// Package buildinfo exposes the provenance injected into Orchestra binaries.
|
||||
// Build systems should set Revision, Time, and Dirty with -ldflags. Keeping
|
||||
// the defaults explicit makes development binaries honest rather than
|
||||
// pretending to be a deployable revision.
|
||||
package buildinfo
|
||||
|
||||
var (
|
||||
Revision = "devel"
|
||||
Time = "unknown"
|
||||
Dirty = "unknown"
|
||||
)
|
||||
|
||||
type Info struct {
|
||||
Revision string `json:"revision"`
|
||||
Time string `json:"time"`
|
||||
Dirty string `json:"dirty"`
|
||||
}
|
||||
|
||||
func Current() Info { return Info{Revision: Revision, Time: Time, Dirty: Dirty} }
|
||||
@@ -22,11 +22,11 @@ var ErrInvalid = errors.New("invalid event")
|
||||
// task for this content; nothing was appended.
|
||||
var ErrDuplicate = errors.New("duplicate task ingestion")
|
||||
|
||||
// CurrentEventSchema is 2: schema 2 requires every event to declare its
|
||||
// authorizing Surface (see ValidateEvent), enforced at the store append
|
||||
// boundary. Schema 1 events already on disk replay unchanged — tolerant
|
||||
// reader, not upcast (spec open question #2).
|
||||
const CurrentEventSchema = 2
|
||||
// CurrentEventSchema is 3: schema 2 requires every event to declare its
|
||||
// authorizing Surface; schema 3 adds lease fencing epochs. Older events stay
|
||||
// readable so a deployment can recover its existing log before new writes
|
||||
// are emitted (the store derives a non-renewable legacy epoch on replay).
|
||||
const CurrentEventSchema = 3
|
||||
|
||||
type TaskState string
|
||||
|
||||
@@ -102,8 +102,13 @@ type SessionEvidence struct {
|
||||
CheckedAt time.Time `json:"checked_at,omitempty"`
|
||||
}
|
||||
type Lease struct {
|
||||
HarnessID string `json:"harness_id"`
|
||||
Until time.Time `json:"until"`
|
||||
HarnessID string `json:"harness_id"`
|
||||
// Epoch is an opaque fencing token minted for every assignment. Versions
|
||||
// change for ordinary lifecycle events; an epoch changes only when
|
||||
// ownership changes, so an old pane can never become current again after
|
||||
// a release/re-lease cycle.
|
||||
Epoch string `json:"epoch"`
|
||||
Until time.Time `json:"until"`
|
||||
}
|
||||
type Task struct {
|
||||
ID string `json:"id"`
|
||||
@@ -191,7 +196,18 @@ func ValidateEvent(e Event) error {
|
||||
if p == nil {
|
||||
return fmt.Errorf("%w: payload must be an object", ErrInvalid)
|
||||
}
|
||||
return ValidatePayload(e.Type, p)
|
||||
if err := ValidatePayload(e.Type, p); err != nil {
|
||||
return err
|
||||
}
|
||||
if e.SchemaVersion >= 3 {
|
||||
switch e.Type {
|
||||
case "TaskLeased", "TaskLeaseRenewed", "TaskPickupValidated":
|
||||
if v, ok := p["lease_epoch"].(string); !ok || strings.TrimSpace(v) == "" {
|
||||
return fmt.Errorf("%w: lease_epoch required", ErrInvalid)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func ValidateCreated(p map[string]any) error {
|
||||
for _, k := range []string{"source", "external_id", "project"} {
|
||||
|
||||
@@ -131,8 +131,8 @@ func (c Client) Heartbeat(ctx context.Context, health WorkerHealth) error {
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (c Client) Renew(ctx context.Context, taskID string, expectedVersion, ttlSeconds int) error {
|
||||
resp, err := c.request(ctx, http.MethodPost, "/v1/federation/workers/"+url.PathEscape(c.WorkerID)+"/renew", map[string]any{"task_id": taskID, "expected_version": expectedVersion, "ttl_seconds": ttlSeconds})
|
||||
func (c Client) Renew(ctx context.Context, taskID, epoch string, expectedVersion, ttlSeconds int) error {
|
||||
resp, err := c.request(ctx, http.MethodPost, "/v1/federation/workers/"+url.PathEscape(c.WorkerID)+"/renew", map[string]any{"task_id": taskID, "lease_epoch": epoch, "expected_version": expectedVersion, "ttl_seconds": ttlSeconds})
|
||||
if resp != nil {
|
||||
resp.Body.Close()
|
||||
}
|
||||
@@ -176,22 +176,22 @@ func (c Client) PutArtifact(ctx context.Context, b []byte) (string, error) {
|
||||
}
|
||||
return out.Ref, nil
|
||||
}
|
||||
func (c Client) Release(ctx context.Context, taskID, ref, anchor, transactionID string, expectedVersion int, evidence domain.SessionEvidence) error {
|
||||
resp, err := c.request(ctx, http.MethodPost, "/v1/federation/workers/"+url.PathEscape(c.WorkerID)+"/handoff", map[string]any{"task_id": taskID, "handoff_ref": ref, "anchor_sha": anchor, "transaction_id": transactionID, "expected_version": expectedVersion, "session_evidence": evidence})
|
||||
func (c Client) Release(ctx context.Context, taskID, ref, anchor, transactionID, epoch string, expectedVersion int, evidence domain.SessionEvidence) error {
|
||||
resp, err := c.request(ctx, http.MethodPost, "/v1/federation/workers/"+url.PathEscape(c.WorkerID)+"/handoff", map[string]any{"task_id": taskID, "handoff_ref": ref, "anchor_sha": anchor, "transaction_id": transactionID, "lease_epoch": epoch, "expected_version": expectedVersion, "session_evidence": evidence})
|
||||
if resp != nil {
|
||||
resp.Body.Close()
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (c Client) Pickup(ctx context.Context, taskID, ref, anchor, transactionID string, leaseVersion int, evidence domain.SessionEvidence) error {
|
||||
resp, err := c.request(ctx, http.MethodPost, "/v1/federation/workers/"+url.PathEscape(c.WorkerID)+"/pickup", map[string]any{"task_id": taskID, "handoff_ref": ref, "anchor_sha": anchor, "transaction_id": transactionID, "lease_version": leaseVersion, "session_evidence": evidence})
|
||||
func (c Client) Pickup(ctx context.Context, taskID, ref, anchor, transactionID, epoch string, leaseVersion int, evidence domain.SessionEvidence) error {
|
||||
resp, err := c.request(ctx, http.MethodPost, "/v1/federation/workers/"+url.PathEscape(c.WorkerID)+"/pickup", map[string]any{"task_id": taskID, "handoff_ref": ref, "anchor_sha": anchor, "transaction_id": transactionID, "lease_epoch": epoch, "lease_version": leaseVersion, "session_evidence": evidence})
|
||||
if resp != nil {
|
||||
resp.Body.Close()
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (c Client) Complete(ctx context.Context, taskID, reportRef, resultSHA, branch, remote string, expectedVersion int, receipt map[string]any, evidence domain.SessionEvidence) error {
|
||||
resp, err := c.request(ctx, http.MethodPost, "/v1/federation/workers/"+url.PathEscape(c.WorkerID)+"/complete", map[string]any{"task_id": taskID, "handoff_ref": reportRef, "result_sha": resultSHA, "branch": branch, "remote": remote, "expected_version": expectedVersion, "receipt": receipt, "session_evidence": evidence})
|
||||
func (c Client) Complete(ctx context.Context, taskID, reportRef, resultSHA, branch, remote, epoch string, expectedVersion int, receipt map[string]any, evidence domain.SessionEvidence) error {
|
||||
resp, err := c.request(ctx, http.MethodPost, "/v1/federation/workers/"+url.PathEscape(c.WorkerID)+"/complete", map[string]any{"task_id": taskID, "handoff_ref": reportRef, "result_sha": resultSHA, "branch": branch, "remote": remote, "lease_epoch": epoch, "expected_version": expectedVersion, "receipt": receipt, "session_evidence": evidence})
|
||||
if resp != nil {
|
||||
resp.Body.Close()
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"orchestra/internal/buildinfo"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
@@ -15,13 +16,15 @@ var ErrUnknownWorker = errors.New("unknown worker")
|
||||
var ErrUnauthorized = errors.New("worker authentication failed")
|
||||
|
||||
type Worker struct {
|
||||
ID string `json:"id"`
|
||||
Address string `json:"address"`
|
||||
Capacity int `json:"capacity"`
|
||||
LastSeen time.Time `json:"last_seen"`
|
||||
Online bool `json:"online"`
|
||||
Health WorkerHealth `json:"health"`
|
||||
Token string `json:"-"`
|
||||
ID string `json:"id"`
|
||||
Address string `json:"address"`
|
||||
Capacity int `json:"capacity"`
|
||||
SupportedProjects []string `json:"supported_projects"`
|
||||
Build buildinfo.Info `json:"build"`
|
||||
LastSeen time.Time `json:"last_seen"`
|
||||
Online bool `json:"online"`
|
||||
Health WorkerHealth `json:"health"`
|
||||
Token string `json:"-"`
|
||||
}
|
||||
|
||||
// WorkerHealth is reported by the worker that owns the local herdr socket.
|
||||
@@ -86,10 +89,12 @@ type persistedState struct {
|
||||
// is mode 0600, and retaining this binding prevents an arbitrary process from
|
||||
// registering a recovered worker ID and executing its pending approval.
|
||||
type persistedWorker struct {
|
||||
ID string `json:"id"`
|
||||
Address string `json:"address"`
|
||||
Capacity int `json:"capacity"`
|
||||
Token string `json:"token"`
|
||||
ID string `json:"id"`
|
||||
Address string `json:"address"`
|
||||
Capacity int `json:"capacity"`
|
||||
SupportedProjects []string `json:"supported_projects"`
|
||||
Build buildinfo.Info `json:"build"`
|
||||
Token string `json:"token"`
|
||||
}
|
||||
|
||||
// Load restores durable capture/command state. Call this before accepting
|
||||
@@ -123,7 +128,7 @@ func (r *Registry) Load() error {
|
||||
if id == "" || w.ID != id || w.Token == "" {
|
||||
return fmt.Errorf("invalid federation worker %q", id)
|
||||
}
|
||||
r.workers[id] = Worker{ID: w.ID, Address: w.Address, Capacity: w.Capacity, Token: w.Token}
|
||||
r.workers[id] = Worker{ID: w.ID, Address: w.Address, Capacity: w.Capacity, SupportedProjects: w.SupportedProjects, Build: w.Build, Token: w.Token}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -135,7 +140,7 @@ func (r *Registry) persistLocked() error {
|
||||
}
|
||||
workers := make(map[string]persistedWorker, len(r.workers))
|
||||
for id, w := range r.workers {
|
||||
workers[id] = persistedWorker{ID: w.ID, Address: w.Address, Capacity: w.Capacity, Token: w.Token}
|
||||
workers[id] = persistedWorker{ID: w.ID, Address: w.Address, Capacity: w.Capacity, SupportedProjects: w.SupportedProjects, Build: w.Build, Token: w.Token}
|
||||
}
|
||||
b, err := json.Marshal(persistedState{Captures: r.captures, Commands: r.commands, Workers: workers})
|
||||
if err != nil {
|
||||
@@ -145,12 +150,31 @@ func (r *Registry) persistLocked() error {
|
||||
return err
|
||||
}
|
||||
tmp := r.StatePath + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0600); err != nil {
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = f.Write(b); err == nil {
|
||||
err = f.Sync()
|
||||
}
|
||||
if closeErr := f.Close(); err == nil {
|
||||
err = closeErr
|
||||
}
|
||||
if err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(tmp, r.StatePath); err != nil {
|
||||
return err
|
||||
}
|
||||
dir, err := os.Open(filepath.Dir(r.StatePath))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer dir.Close()
|
||||
if err := dir.Sync(); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Chmod(r.StatePath, 0600)
|
||||
}
|
||||
|
||||
@@ -396,10 +420,32 @@ func (r *Registry) Available(id string) bool {
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
w.Online = time.Since(w.LastSeen) <= r.TTL
|
||||
// A heartbeat merely proves the worker process can reach the coordinator.
|
||||
// Lease admission additionally requires a fresh probe of the worker's
|
||||
// local herdr; otherwise a partitioned/down herdr still attracts work.
|
||||
w.Online = time.Since(w.LastSeen) <= r.TTL && w.Health.HerdrStatus == "reachable" && !w.Health.CheckedAt.IsZero() && time.Since(w.Health.CheckedAt) <= r.TTL
|
||||
r.workers[id] = w
|
||||
return w.Online
|
||||
}
|
||||
|
||||
// Supports reports whether an online worker explicitly declared the project.
|
||||
// An omitted declaration is deliberately not treated as a wildcard: workers
|
||||
// must never receive a project for which they have no local checkout.
|
||||
func (r *Registry) Supports(id, project string) bool {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.init()
|
||||
w, ok := r.workers[id]
|
||||
if !ok || time.Since(w.LastSeen) > r.TTL {
|
||||
return false
|
||||
}
|
||||
for _, candidate := range w.SupportedProjects {
|
||||
if candidate == project {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
func (r *Registry) Snapshot() []Worker {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
@@ -29,6 +29,49 @@ func TestCursorIsMonotonicAndAuthenticationIsRequired(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSupportedProjectsPersistAndGateAvailability(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "state.json")
|
||||
r := &Registry{StatePath: path}
|
||||
if err := r.Register(Worker{ID: "w", Token: "t", SupportedProjects: []string{"test-e2e"}}, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !r.Supports("w", "test-e2e") || r.Supports("w", "correx") {
|
||||
t.Fatalf("unexpected project support")
|
||||
}
|
||||
restarted := &Registry{StatePath: path}
|
||||
if err := restarted.Load(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := restarted.Register(Worker{ID: "w", Token: "t", SupportedProjects: []string{"test-e2e"}}, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !restarted.Supports("w", "test-e2e") || restarted.Supports("w", "correx") {
|
||||
t.Fatalf("project support did not survive restart")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAvailableRequiresFreshReachableLocalHerdrHealth(t *testing.T) {
|
||||
r := &Registry{}
|
||||
if err := r.Register(Worker{ID: "w", Token: "t"}, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r.Available("w") {
|
||||
t.Fatal("registration without local herdr probe admitted a worker")
|
||||
}
|
||||
if err := r.Heartbeat("w", WorkerHealth{HerdrStatus: "reachable", CheckedAt: time.Now().UTC()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !r.Available("w") {
|
||||
t.Fatal("fresh reachable local herdr was not admitted")
|
||||
}
|
||||
if err := r.Heartbeat("w", WorkerHealth{HerdrStatus: "unreachable", CheckedAt: time.Now().UTC()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r.Available("w") {
|
||||
t.Fatal("unreachable local herdr was admitted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPendingApprovalSurvivesRegistryRestart(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "federation-state.json")
|
||||
r := &Registry{StatePath: path}
|
||||
|
||||
@@ -129,9 +129,12 @@ func TestEndToEndIngestRouteLeaseRotateAndComplete(t *testing.T) {
|
||||
}
|
||||
got, _ := s.Task(task.ID)
|
||||
if got.State == domain.StateLeased && h.releases == 1 {
|
||||
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: got.Version + 1, Surface: string(authz.System), Payload: mustJSON(map[string]string{
|
||||
"handoff_ref": h.ref,
|
||||
"anchor_sha": "0123456789012345678901234567890123456789",
|
||||
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: got.Version + 1, Surface: string(authz.System), Payload: mustJSON(map[string]any{
|
||||
"handoff_ref": h.ref,
|
||||
"anchor_sha": "0123456789012345678901234567890123456789",
|
||||
"harness_id": got.Lease.HarnessID,
|
||||
"lease_epoch": got.Lease.Epoch,
|
||||
"expected_version": got.Version,
|
||||
})}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -186,9 +189,13 @@ func TestRestartReplayAndReconcileKillsOrphan(t *testing.T) {
|
||||
}
|
||||
// A fresh coordinator sees the durable session, then drops it once the lease is gone.
|
||||
ref, _ := s.PutArtifact([]byte("handoff"))
|
||||
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: 3, Surface: string(authz.System), Payload: mustJSON(map[string]string{
|
||||
"handoff_ref": ref,
|
||||
"anchor_sha": "0123456789012345678901234567890123456789",
|
||||
leased, _ := s.Task(task.ID)
|
||||
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: 3, Surface: string(authz.System), Payload: mustJSON(map[string]any{
|
||||
"handoff_ref": ref,
|
||||
"anchor_sha": "0123456789012345678901234567890123456789",
|
||||
"harness_id": leased.Lease.HarnessID,
|
||||
"lease_epoch": leased.Lease.Epoch,
|
||||
"expected_version": leased.Version,
|
||||
})}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -230,9 +237,13 @@ func TestProviderRetryReflectionQuotaAndVersionConflict(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reflector := &fakeReflector{}
|
||||
leased, _ := s.Task(task.ID)
|
||||
if err := (provider.ReflectingSink{Sink: s, Tasks: s, Reflector: reflector}).Append(domain.Event{ID: domain.NewID(), Type: "TaskCompleted", TaskID: task.ID, Version: 3, Surface: string(authz.System), Payload: mustJSON(map[string]any{
|
||||
"report_ref": ref,
|
||||
"receipt": map[string]any{"harness_id": "h1", "consumed": 1},
|
||||
"report_ref": ref,
|
||||
"receipt": map[string]any{"harness_id": "h1", "consumed": 1},
|
||||
"harness_id": leased.Lease.HarnessID,
|
||||
"lease_epoch": leased.Lease.Epoch,
|
||||
"expected_version": leased.Version,
|
||||
})}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -6,6 +6,7 @@ package orchestrator
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"orchestra/internal/authz"
|
||||
"orchestra/internal/continuity"
|
||||
@@ -412,10 +413,29 @@ func (c *Coordinator) saveSessionsLocked() error {
|
||||
return err
|
||||
}
|
||||
tmp := c.StatePath + ".tmp"
|
||||
if err = os.WriteFile(tmp, b, 0600); err != nil {
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, c.StatePath)
|
||||
if _, err = f.Write(b); err == nil {
|
||||
err = f.Sync()
|
||||
}
|
||||
if closeErr := f.Close(); err == nil {
|
||||
err = closeErr
|
||||
}
|
||||
if err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err = os.Rename(tmp, c.StatePath); err != nil {
|
||||
return err
|
||||
}
|
||||
dir, err := os.Open(filepath.Dir(c.StatePath))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer dir.Close()
|
||||
return dir.Sync()
|
||||
}
|
||||
|
||||
// Reconcile drops mappings whose task lease did not survive restart and kills
|
||||
@@ -573,7 +593,7 @@ func (c *Coordinator) expire(ctx context.Context) ([]domain.Event, error) {
|
||||
if a, ae := c.adapterFor(taskID, s); ae == nil {
|
||||
if p, ok := a.(herdr.PaneExit); ok {
|
||||
if exited, ee := p.PaneExited(ctx, s); ee == nil && exited {
|
||||
b, _ := json.Marshal(map[string]string{"reason": "pane_exited", "harness_id": s.Harness})
|
||||
b, _ := json.Marshal(map[string]any{"reason": "pane_exited", "harness_id": s.Harness, "lease_epoch": t.Lease.Epoch, "expected_version": t.Version})
|
||||
_ = c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: t.Version + 1, Payload: b, Surface: string(authz.System)})
|
||||
}
|
||||
}
|
||||
@@ -581,24 +601,47 @@ func (c *Coordinator) expire(ctx context.Context) ([]domain.Event, error) {
|
||||
}
|
||||
}
|
||||
c.mu.Unlock()
|
||||
events, err := c.Store.ExpireLeases(time.Now())
|
||||
if err != nil {
|
||||
return events, err
|
||||
}
|
||||
for _, e := range events {
|
||||
c.loadSessions()
|
||||
c.mu.Lock()
|
||||
s, ok := c.sessions[e.TaskID]
|
||||
delete(c.sessions, e.TaskID)
|
||||
if ok {
|
||||
if a, ae := c.adapterFor(e.TaskID, s); ae == nil {
|
||||
_ = a.Kill(ctx, s)
|
||||
}
|
||||
var events []domain.Event
|
||||
var firstErr error
|
||||
for _, task := range c.Store.Tasks() {
|
||||
if task.State != domain.StateLeased || task.Lease == nil || task.Lease.Until.After(time.Now()) {
|
||||
continue
|
||||
}
|
||||
_ = c.saveSessionsLocked()
|
||||
// Stop a local predecessor before making its lease eligible for a
|
||||
// successor. If this cannot be done, keep both the mapping and the
|
||||
// lease: safety beats reclaim speed.
|
||||
c.mu.Lock()
|
||||
s, local := c.sessions[task.ID]
|
||||
c.mu.Unlock()
|
||||
if local {
|
||||
a, adapterErr := c.adapterFor(task.ID, s)
|
||||
if adapterErr != nil {
|
||||
if firstErr == nil {
|
||||
firstErr = fmt.Errorf("expire %s: resolve old pane: %w", task.ID, adapterErr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if killErr := a.Kill(ctx, s); killErr != nil {
|
||||
if firstErr == nil {
|
||||
firstErr = fmt.Errorf("expire %s: quarantine old pane: %w", task.ID, killErr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
c.mu.Lock()
|
||||
delete(c.sessions, task.ID)
|
||||
_ = c.saveSessionsLocked()
|
||||
c.mu.Unlock()
|
||||
}
|
||||
e, expireErr := c.Store.ExpireLease(task.ID, time.Now())
|
||||
if expireErr != nil {
|
||||
if !errors.Is(expireErr, domain.ErrConflict) && firstErr == nil {
|
||||
firstErr = expireErr
|
||||
}
|
||||
continue
|
||||
}
|
||||
events = append(events, e)
|
||||
}
|
||||
return events, nil
|
||||
return events, firstErr
|
||||
}
|
||||
|
||||
// handoffReason reads HandoffFile from the worktree, if present, and returns
|
||||
@@ -722,13 +765,17 @@ func (c *Coordinator) rotate(ctx context.Context, hard float64) {
|
||||
// tick or TTL expiry to reclaim.
|
||||
continue
|
||||
}
|
||||
b, _ := json.Marshal(map[string]string{"handoff_ref": ref, "reason": reason, "anchor_sha": anchorSHA})
|
||||
b, _ := json.Marshal(map[string]any{"handoff_ref": ref, "reason": reason, "anchor_sha": anchorSHA, "harness_id": task.Lease.HarnessID, "lease_epoch": task.Lease.Epoch, "expected_version": task.Version})
|
||||
e := domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: task.Version + 1, Payload: b, Surface: string(authz.System)}
|
||||
if c.Store.Append(e) == nil {
|
||||
c.mu.Lock()
|
||||
delete(c.sessions, taskID)
|
||||
_ = c.saveSessionsLocked()
|
||||
c.mu.Unlock()
|
||||
// A release only transfers the lease; this local coordinator owns
|
||||
// the predecessor pane until it has actually stopped it.
|
||||
if err := a.Kill(ctx, session); err == nil {
|
||||
c.mu.Lock()
|
||||
delete(c.sessions, taskID)
|
||||
_ = c.saveSessionsLocked()
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -842,11 +889,14 @@ func (c *Coordinator) finishRelease(ctx context.Context, taskID string, task dom
|
||||
// invalid TaskReleased payload, same as rotate()'s bare continue.
|
||||
return TurnRefuse, nil
|
||||
}
|
||||
b, _ := json.Marshal(map[string]string{"handoff_ref": ref, "reason": reason, "anchor_sha": anchorSHA})
|
||||
b, _ := json.Marshal(map[string]any{"handoff_ref": ref, "reason": reason, "anchor_sha": anchorSHA, "harness_id": task.Lease.HarnessID, "lease_epoch": task.Lease.Epoch, "expected_version": task.Version})
|
||||
e := domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: task.Version + 1, Payload: b, Surface: string(authz.System)}
|
||||
if err := c.Store.Append(e); err != nil {
|
||||
return TurnRefuse, nil
|
||||
}
|
||||
if err := a.Kill(ctx, session); err != nil {
|
||||
return TurnRefuse, nil
|
||||
}
|
||||
c.mu.Lock()
|
||||
delete(c.sessions, taskID)
|
||||
_ = c.saveSessionsLocked()
|
||||
@@ -978,6 +1028,12 @@ func (c *Coordinator) block(t domain.Task, reason string) error {
|
||||
p["session_evidence"] = domain.SessionEvidence{PaneID: s.PaneID, HarnessID: s.HerdrID, PaneState: "open", Source: "coordinator", CheckedAt: time.Now().UTC()}
|
||||
}
|
||||
b, _ := json.Marshal(p)
|
||||
if t.Lease != nil {
|
||||
p["harness_id"] = t.Lease.HarnessID
|
||||
p["lease_epoch"] = t.Lease.Epoch
|
||||
p["expected_version"] = t.Version
|
||||
b, _ = json.Marshal(p)
|
||||
}
|
||||
return c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskBlocked", TaskID: t.ID, Version: t.Version + 1, Payload: b, Surface: string(authz.System)})
|
||||
}
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@@ -73,7 +74,7 @@ func TestGitWorktreesCommitsTaskFile(t *testing.T) {
|
||||
initRepo(t, repo)
|
||||
|
||||
w := orchestrator.GitWorktrees{Root: filepath.Join(base, "wt"), Repo: repo}
|
||||
task := domain.Task{ID: "t1", Project: "p", Source: "jsonl", ExternalID: "1", Title: "do the thing"}
|
||||
task := domain.Task{ID: "t1", Project: "p", Source: "jsonl", ExternalID: "1", Title: "do the thing", Acceptance: []string{"tests pass"}, QualityGate: "go test ./..."}
|
||||
|
||||
path, err := w.Create(context.Background(), task)
|
||||
if err != nil {
|
||||
@@ -88,6 +89,9 @@ func TestGitWorktreesCommitsTaskFile(t *testing.T) {
|
||||
if string(got) != string(want) {
|
||||
t.Fatalf("TASK.md content mismatch:\ngot: %s\nwant: %s", got, want)
|
||||
}
|
||||
if !strings.Contains(string(got), "## Acceptance criteria") || !strings.Contains(string(got), ".orchestra/done") {
|
||||
t.Fatalf("TASK.md is missing deterministic completion contract: %s", got)
|
||||
}
|
||||
|
||||
status, err := exec.Command("git", "-C", path, "status", "--porcelain", "--", "TASK.md").Output()
|
||||
if err != nil {
|
||||
|
||||
@@ -28,6 +28,7 @@ type Project struct {
|
||||
// wires (single-repo deployments keep working unchanged).
|
||||
Repo string `json:"repo,omitempty"`
|
||||
WorktreeRoot string `json:"worktree_root,omitempty"`
|
||||
QualityGate string `json:"quality_gate,omitempty"`
|
||||
}
|
||||
type Machine struct {
|
||||
ID string `json:"id"`
|
||||
|
||||
@@ -14,6 +14,12 @@ import (
|
||||
)
|
||||
|
||||
type Availability interface{ Available(h registry.Herdr) bool }
|
||||
|
||||
// ProjectAvailability is an optional stricter availability contract used by
|
||||
// federated workers, whose local checkout configuration is authoritative.
|
||||
type ProjectAvailability interface {
|
||||
Supports(h registry.Herdr, project string) bool
|
||||
}
|
||||
type AlwaysAvailable struct{}
|
||||
|
||||
func (AlwaysAvailable) Available(registry.Herdr) bool { return true }
|
||||
@@ -170,7 +176,11 @@ func (r *Router) AssignPending() ([]domain.Event, error) {
|
||||
continue
|
||||
}
|
||||
for _, h := range cs {
|
||||
if !matches(t.Capability, h.Capabilities) || !r.Availability.Available(h) || occupied(r.Store, h.ID, h.Concurrency) {
|
||||
projectOK := true
|
||||
if projects, ok := r.Availability.(ProjectAvailability); ok {
|
||||
projectOK = projects.Supports(h, t.Project)
|
||||
}
|
||||
if !projectOK || !matches(t.Capability, h.Capabilities) || !r.Availability.Available(h) || occupied(r.Store, h.ID, h.Concurrency) {
|
||||
continue
|
||||
}
|
||||
e, err := r.Store.Lease(t.ID, h.ID, 30*time.Minute)
|
||||
|
||||
@@ -14,6 +14,13 @@ type reachable struct{}
|
||||
|
||||
func (reachable) Reachable(string, time.Duration) bool { return true }
|
||||
|
||||
type projectAvailability struct{ projects map[string]bool }
|
||||
|
||||
func (p projectAvailability) Available(registry.Herdr) bool { return true }
|
||||
func (p projectAvailability) Supports(h registry.Herdr, project string) bool {
|
||||
return p.projects[h.ID+"/"+project]
|
||||
}
|
||||
|
||||
func TestAssignsByAffinityCapabilityAndConcurrency(t *testing.T) {
|
||||
s, err := store.Open(t.TempDir())
|
||||
if err != nil {
|
||||
@@ -56,6 +63,32 @@ func TestAssignsByAffinityCapabilityAndConcurrency(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssignPendingRequiresWorkerProjectSupport(t *testing.T) {
|
||||
s, err := store.Open(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, err := registry.New(registry.Config{Projects: []registry.Project{{ID: "p", MachineAffinity: []string{"m"}}}, Machines: []registry.Machine{{ID: "m", Address: "unused"}}, Herdrs: []registry.Herdr{{ID: "h", MachineID: "m", Concurrency: 1}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := json.Marshal(map[string]any{"source": "test", "external_id": "project-check", "project": "p"})
|
||||
if err := s.Append(domain.Event{ID: "create", TaskID: "t", Type: "TaskCreated", Version: 1, Payload: b, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rt := Router{Store: s, Registry: r, Reachability: reachable{}, Availability: projectAvailability{projects: map[string]bool{}}}
|
||||
if got, err := rt.AssignPending(); err != nil || len(got) != 0 {
|
||||
t.Fatalf("unsupported project lease = %#v, %v", got, err)
|
||||
}
|
||||
if got, _ := s.Task("t"); got.State != domain.StateQueued {
|
||||
t.Fatalf("unsupported project state=%s", got.State)
|
||||
}
|
||||
rt.Availability = projectAvailability{projects: map[string]bool{"h/p": true}}
|
||||
if got, err := rt.AssignPending(); err != nil || len(got) != 1 {
|
||||
t.Fatalf("supported project lease = %#v, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRotationDoesNotCountAgainstRetryLimit guards B4: rotation is
|
||||
// TaskReleased carrying a valid handoff_ref (spec §5.3: "rotation =
|
||||
// intra-task lease transfer"), never a failure. A task healthy enough to
|
||||
@@ -103,10 +136,13 @@ func TestRotationDoesNotCountAgainstRetryLimit(t *testing.T) {
|
||||
}
|
||||
continue
|
||||
}
|
||||
rb, _ := json.Marshal(map[string]string{
|
||||
"handoff_ref": handoffRef,
|
||||
"reason": "threshold",
|
||||
"anchor_sha": "0123456789abcdef0123456789abcdef01234567",
|
||||
rb, _ := json.Marshal(map[string]any{
|
||||
"handoff_ref": handoffRef,
|
||||
"reason": "threshold",
|
||||
"anchor_sha": "0123456789abcdef0123456789abcdef01234567",
|
||||
"harness_id": task.Lease.HarnessID,
|
||||
"lease_epoch": task.Lease.Epoch,
|
||||
"expected_version": task.Version,
|
||||
})
|
||||
release := domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: "a", Version: task.Version + 1, Payload: rb, Surface: string(authz.System)}
|
||||
if err := s.Append(release); err != nil {
|
||||
|
||||
+148
-43
@@ -32,27 +32,10 @@ func Open(dir string) (*Store, error) {
|
||||
if err := os.MkdirAll(s.cas, 0755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var snapshotSeq uint64
|
||||
if b, readErr := os.ReadFile(s.snapshot); readErr == nil {
|
||||
var snap struct {
|
||||
Seq uint64 `json:"seq"`
|
||||
Tasks []domain.Task `json:"tasks"`
|
||||
}
|
||||
if json.Unmarshal(b, &snap) != nil {
|
||||
return nil, fmt.Errorf("invalid snapshot")
|
||||
}
|
||||
for _, t := range snap.Tasks {
|
||||
s.tasks[t.ID] = t
|
||||
s.external[t.Source+"\x00"+t.ExternalID] = t.ID
|
||||
}
|
||||
snapshotSeq = snap.Seq
|
||||
// Continue event numbering after the snapshot. Without restoring this
|
||||
// cursor, the first append after a restart reused sequence 1 and made
|
||||
// the append-only log unreplayable.
|
||||
s.seq = snapshotSeq
|
||||
} else if !errors.Is(readErr, os.ErrNotExist) {
|
||||
return nil, readErr
|
||||
}
|
||||
// A snapshot is a disposable read cache, never recovery authority. Loading
|
||||
// it before the log let a partially-written snapshot become a different
|
||||
// history than events.jsonl after a crash. Rebuild every projection from
|
||||
// the append-only, fsynced log instead.
|
||||
f, err := os.Open(s.path)
|
||||
if os.IsNotExist(err) {
|
||||
return s, nil
|
||||
@@ -62,16 +45,13 @@ func Open(dir string) (*Store, error) {
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
var expected uint64 = snapshotSeq + 1
|
||||
var expected uint64 = 1
|
||||
for sc.Scan() {
|
||||
var e domain.Event
|
||||
if err := json.Unmarshal(sc.Bytes(), &e); err == nil {
|
||||
if err := domain.ValidateEvent(e); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if e.Seq < expected {
|
||||
continue
|
||||
}
|
||||
if e.Seq != expected {
|
||||
return nil, fmt.Errorf("event sequence gap: got %d, want %d", e.Seq, expected)
|
||||
}
|
||||
@@ -151,9 +131,20 @@ func (s *Store) apply(e domain.Event) error {
|
||||
s.external[t.Source+"\x00"+t.ExternalID] = t.ID
|
||||
case "TaskLeased":
|
||||
t.State = domain.StateLeased
|
||||
t.Lease = &domain.Lease{HarnessID: p["harness_id"].(string), Until: time.Unix(0, int64(p["until_ns"].(float64)))}
|
||||
epoch, _ := p["lease_epoch"].(string)
|
||||
if epoch == "" {
|
||||
// A pre-fencing event cannot safely be renewed by an old worker.
|
||||
// Deriving a stable token from the durable event identity makes the
|
||||
// recovered lease observable but non-renewable until it expires.
|
||||
epoch = "legacy:" + e.ID
|
||||
}
|
||||
t.Lease = &domain.Lease{HarnessID: p["harness_id"].(string), Epoch: epoch, Until: time.Unix(0, int64(p["until_ns"].(float64)))}
|
||||
case "TaskLeaseRenewed":
|
||||
t.Lease = &domain.Lease{HarnessID: p["harness_id"].(string), Until: time.Unix(0, int64(p["until_ns"].(float64)))}
|
||||
epoch, _ := p["lease_epoch"].(string)
|
||||
if epoch == "" && t.Lease != nil {
|
||||
epoch = t.Lease.Epoch
|
||||
}
|
||||
t.Lease = &domain.Lease{HarnessID: p["harness_id"].(string), Epoch: epoch, Until: time.Unix(0, int64(p["until_ns"].(float64)))}
|
||||
case "TaskReleased":
|
||||
t.State = domain.StateQueued
|
||||
t.Lease = nil
|
||||
@@ -292,6 +283,9 @@ func (s *Store) Append(e domain.Event) error {
|
||||
return domain.ErrConflict
|
||||
}
|
||||
}
|
||||
if err := s.validateTransition(e, t, taskExists, contract); err != nil {
|
||||
return err
|
||||
}
|
||||
if e.Type == "TaskLeased" {
|
||||
var p struct {
|
||||
ExpectedVersion *int `json:"expected_version"`
|
||||
@@ -333,9 +327,6 @@ func (s *Store) Append(e domain.Event) error {
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := s.apply(e); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.OpenFile(s.path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0644)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -348,10 +339,60 @@ func (s *Store) Append(e domain.Event) error {
|
||||
if err = f.Sync(); err != nil {
|
||||
return err
|
||||
}
|
||||
// The event is the commit record. Do not expose a projection that cannot
|
||||
// be recovered from it after a power loss.
|
||||
if err := s.apply(e); err != nil {
|
||||
return err
|
||||
}
|
||||
s.events = append(s.events, e)
|
||||
s.seq = e.Seq
|
||||
if err := s.writeSnapshot(); err != nil {
|
||||
return err
|
||||
// Snapshot failure does not roll back a committed event. Open always
|
||||
// rebuilds from the log, so leaving a stale cache is safe.
|
||||
_ = s.writeSnapshot()
|
||||
return nil
|
||||
}
|
||||
|
||||
// validateTransition keeps lifecycle authority at the durable append
|
||||
// boundary. A task may be completed/failed while queued by an external
|
||||
// provider, but once a lease exists its owner and fencing epoch are required
|
||||
// for every lifecycle mutation.
|
||||
func (s *Store) validateTransition(e domain.Event, t domain.Task, exists bool, p map[string]any) error {
|
||||
if !exists {
|
||||
if e.Type != "TaskCreated" && e.Type != "QuotaReported" && e.Type != "StandupAdvisory" && e.Type != "ApprovalGranted" && e.Type != "ApprovalDenied" {
|
||||
return domain.ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if e.Type == "TaskLeased" && t.State != domain.StateQueued {
|
||||
return domain.ErrConflict
|
||||
}
|
||||
if t.State != domain.StateLeased || t.Lease == nil {
|
||||
return nil
|
||||
}
|
||||
switch e.Type {
|
||||
case "TaskLeaseRenewed", "TaskReleased", "TaskPickupValidated", "TaskCompleted", "TaskBlocked", "TaskFailed":
|
||||
owner, _ := p["harness_id"].(string)
|
||||
epoch, _ := p["lease_epoch"].(string)
|
||||
// Expiry is the one coordinator-owned relinquish path. It still binds
|
||||
// the exact epoch that was observed when the timer fired.
|
||||
if e.Type == "TaskReleased" {
|
||||
if reason, _ := p["reason"].(string); (reason == "lease_expired" || reason == "pane_exited") && owner == t.Lease.HarnessID && epoch == t.Lease.Epoch {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if owner != t.Lease.HarnessID || epoch == "" || epoch != t.Lease.Epoch {
|
||||
return domain.ErrConflict
|
||||
}
|
||||
case "TaskCorrected":
|
||||
// Corrections may repair metadata while a task is leased, but cannot
|
||||
// smuggle in a lifecycle transition around the current fenced owner.
|
||||
if _, changesState := p["state"]; changesState {
|
||||
owner, _ := p["harness_id"].(string)
|
||||
epoch, _ := p["lease_epoch"].(string)
|
||||
if owner != t.Lease.HarnessID || epoch == "" || epoch != t.Lease.Epoch {
|
||||
return domain.ErrConflict
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -368,10 +409,28 @@ func (s *Store) writeSnapshot() error {
|
||||
return err
|
||||
}
|
||||
tmp := s.snapshot + ".tmp"
|
||||
if err = os.WriteFile(tmp, b, 0644); err != nil {
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, s.snapshot)
|
||||
if _, err = f.Write(b); err == nil {
|
||||
err = f.Sync()
|
||||
}
|
||||
if closeErr := f.Close(); err == nil {
|
||||
err = closeErr
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = os.Rename(tmp, s.snapshot); err != nil {
|
||||
return err
|
||||
}
|
||||
dir, err := os.Open(filepath.Dir(s.snapshot))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer dir.Close()
|
||||
return dir.Sync()
|
||||
}
|
||||
func (s *Store) Tasks() []domain.Task {
|
||||
s.mu.Lock()
|
||||
@@ -397,9 +456,39 @@ func (s *Store) PutArtifact(b []byte) (string, error) {
|
||||
h := domain.Hash(b)
|
||||
p := filepath.Join(s.cas, h)
|
||||
if _, err := os.Stat(p); errors.Is(err, os.ErrNotExist) {
|
||||
if err = os.WriteFile(p, b, 0644); err != nil {
|
||||
tmp := p + ".tmp-" + domain.NewID()
|
||||
f, openErr := os.OpenFile(tmp, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0644)
|
||||
if openErr != nil {
|
||||
return "", openErr
|
||||
}
|
||||
if _, err = f.Write(b); err == nil {
|
||||
err = f.Sync()
|
||||
}
|
||||
if closeErr := f.Close(); err == nil {
|
||||
err = closeErr
|
||||
}
|
||||
if err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return "", err
|
||||
}
|
||||
if err = os.Rename(tmp, p); err != nil && !errors.Is(err, os.ErrExist) {
|
||||
_ = os.Remove(tmp)
|
||||
return "", err
|
||||
}
|
||||
if !errors.Is(err, os.ErrExist) {
|
||||
dir, openErr := os.Open(s.cas)
|
||||
if openErr != nil {
|
||||
return "", openErr
|
||||
}
|
||||
syncErr := dir.Sync()
|
||||
closeErr := dir.Close()
|
||||
if syncErr != nil {
|
||||
return "", syncErr
|
||||
}
|
||||
if closeErr != nil {
|
||||
return "", closeErr
|
||||
}
|
||||
}
|
||||
}
|
||||
return h, nil
|
||||
}
|
||||
@@ -450,7 +539,7 @@ func (s *Store) Lease(id, harness string, ttl time.Duration) (domain.Event, erro
|
||||
if t.State != domain.StateQueued {
|
||||
return domain.Event{}, domain.ErrConflict
|
||||
}
|
||||
payload := map[string]any{"harness_id": harness, "ttl": ttl.Seconds(), "until_ns": time.Now().Add(ttl).UnixNano(), "expected_version": t.Version}
|
||||
payload := map[string]any{"harness_id": harness, "lease_epoch": domain.NewID(), "ttl": ttl.Seconds(), "until_ns": time.Now().Add(ttl).UnixNano(), "expected_version": t.Version}
|
||||
if t.HandoffRef != "" {
|
||||
payload["handoff_ref"] = t.HandoffRef
|
||||
payload["transaction_id"] = t.ReleaseTransaction
|
||||
@@ -464,7 +553,7 @@ func (s *Store) Lease(id, harness string, ttl time.Duration) (domain.Event, erro
|
||||
// RenewLease atomically extends the current owner's lease. The observed task
|
||||
// version is part of the request so an old worker can never renew a lease
|
||||
// after release/reassignment.
|
||||
func (s *Store) RenewLease(id, harness string, expectedVersion int, ttl time.Duration) (domain.Event, error) {
|
||||
func (s *Store) RenewLease(id, harness, epoch string, expectedVersion int, ttl time.Duration) (domain.Event, error) {
|
||||
if ttl <= 0 {
|
||||
return domain.Event{}, fmt.Errorf("%w: ttl must be positive", domain.ErrInvalid)
|
||||
}
|
||||
@@ -472,10 +561,10 @@ func (s *Store) RenewLease(id, harness string, expectedVersion int, ttl time.Dur
|
||||
if !ok {
|
||||
return domain.Event{}, domain.ErrNotFound
|
||||
}
|
||||
if t.State != domain.StateLeased || t.Lease == nil || t.Lease.HarnessID != harness || t.Version != expectedVersion {
|
||||
if t.State != domain.StateLeased || t.Lease == nil || t.Lease.HarnessID != harness || t.Lease.Epoch != epoch || t.Version != expectedVersion {
|
||||
return domain.Event{}, domain.ErrConflict
|
||||
}
|
||||
p, _ := json.Marshal(map[string]any{"harness_id": harness, "until_ns": time.Now().Add(ttl).UnixNano(), "expected_version": expectedVersion})
|
||||
p, _ := json.Marshal(map[string]any{"harness_id": harness, "lease_epoch": epoch, "until_ns": time.Now().Add(ttl).UnixNano(), "expected_version": expectedVersion})
|
||||
e := domain.Event{ID: domain.NewID(), Type: "TaskLeaseRenewed", TaskID: id, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
|
||||
return e, s.Append(e)
|
||||
}
|
||||
@@ -483,14 +572,30 @@ func (s *Store) RenewLease(id, harness string, expectedVersion int, ttl time.Dur
|
||||
func (s *Store) ExpireLeases(now time.Time) ([]domain.Event, error) {
|
||||
var out []domain.Event
|
||||
for _, t := range s.Tasks() {
|
||||
if t.State == domain.StateLeased && t.Lease != nil && !t.Lease.Until.After(now) {
|
||||
p, _ := json.Marshal(map[string]any{"reason": "lease_expired", "harness_id": t.Lease.HarnessID})
|
||||
e := domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
|
||||
if err := s.Append(e); err != nil {
|
||||
if e, err := s.ExpireLease(t.ID, now); err != nil {
|
||||
if !errors.Is(err, domain.ErrConflict) {
|
||||
return out, err
|
||||
}
|
||||
} else if e.ID != "" {
|
||||
out = append(out, e)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ExpireLease releases exactly the observed lease if, and only if, its TTL
|
||||
// has elapsed. Coordinators use this one-task form to stop their local pane
|
||||
// before publishing the release event; the batch helper remains for
|
||||
// deployments without a local coordinator.
|
||||
func (s *Store) ExpireLease(id string, now time.Time) (domain.Event, error) {
|
||||
t, ok := s.Task(id)
|
||||
if !ok {
|
||||
return domain.Event{}, domain.ErrNotFound
|
||||
}
|
||||
if t.State != domain.StateLeased || t.Lease == nil || t.Lease.Until.After(now) {
|
||||
return domain.Event{}, domain.ErrConflict
|
||||
}
|
||||
p, _ := json.Marshal(map[string]any{"reason": "lease_expired", "harness_id": t.Lease.HarnessID, "lease_epoch": t.Lease.Epoch, "expected_version": t.Version})
|
||||
e := domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
|
||||
return e, s.Append(e)
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package store
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -35,7 +36,7 @@ func TestLeaseCarriesReleasedHandoffRef(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ := s.Task("t")
|
||||
p, _ := json.Marshal(map[string]string{"handoff_ref": ref, "anchor_sha": "0123456789012345678901234567890123456789"})
|
||||
p, _ := json.Marshal(map[string]any{"handoff_ref": ref, "anchor_sha": "0123456789012345678901234567890123456789", "harness_id": task.Lease.HarnessID, "lease_epoch": task.Lease.Epoch, "expected_version": task.Version})
|
||||
if err := s.Append(domain.Event{ID: "release", Type: "TaskReleased", TaskID: "t", Version: task.Version + 1, Payload: p, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -64,13 +65,13 @@ func TestRenewLeaseRequiresCurrentOwnerAndVersion(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
before, _ := s.Task("t")
|
||||
if _, err := s.RenewLease("t", "worker-b", before.Version, time.Hour); !errors.Is(err, domain.ErrConflict) {
|
||||
if _, err := s.RenewLease("t", "worker-b", before.Lease.Epoch, before.Version, time.Hour); !errors.Is(err, domain.ErrConflict) {
|
||||
t.Fatalf("other worker renewal = %v, want conflict", err)
|
||||
}
|
||||
if _, err := s.RenewLease("t", "worker-a", before.Version-1, time.Hour); !errors.Is(err, domain.ErrConflict) {
|
||||
if _, err := s.RenewLease("t", "worker-a", before.Lease.Epoch, before.Version-1, time.Hour); !errors.Is(err, domain.ErrConflict) {
|
||||
t.Fatalf("stale renewal = %v, want conflict", err)
|
||||
}
|
||||
e, err := s.RenewLease("t", "worker-a", before.Version, time.Hour)
|
||||
e, err := s.RenewLease("t", "worker-a", before.Lease.Epoch, before.Version, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -78,11 +79,107 @@ func TestRenewLeaseRequiresCurrentOwnerAndVersion(t *testing.T) {
|
||||
if e.Type != "TaskLeaseRenewed" || after.Version != before.Version+1 || after.Lease == nil || !after.Lease.Until.After(before.Lease.Until) {
|
||||
t.Fatalf("renewal was not projected: before=%+v after=%+v event=%+v", before, after, e)
|
||||
}
|
||||
if _, err := s.RenewLease("t", "worker-a", before.Version, time.Hour); !errors.Is(err, domain.ErrConflict) {
|
||||
if _, err := s.RenewLease("t", "worker-a", before.Lease.Epoch, before.Version, time.Hour); !errors.Is(err, domain.ErrConflict) {
|
||||
t.Fatalf("replayed renewal = %v, want conflict", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLeaseEpochFencesStaleOwnerLifecycleWrites(t *testing.T) {
|
||||
s, err := Open(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Append(domain.Event{ID: "create", Type: "TaskCreated", TaskID: "t", Version: 1, Payload: []byte(`{"source":"s","external_id":"epoch","project":"p"}`), Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Lease("t", "worker", time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
first, _ := s.Task("t")
|
||||
ref, err := s.PutArtifact([]byte("handoff"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
staleRelease, _ := json.Marshal(map[string]any{"handoff_ref": ref, "anchor_sha": strings.Repeat("a", 40), "harness_id": "worker", "lease_epoch": "stale", "expected_version": first.Version})
|
||||
if err := s.Append(domain.Event{ID: "stale-release", Type: "TaskReleased", TaskID: "t", Version: first.Version + 1, Payload: staleRelease, Surface: string(authz.System)}); !errors.Is(err, domain.ErrConflict) {
|
||||
t.Fatalf("stale release = %v, want conflict", err)
|
||||
}
|
||||
release, _ := json.Marshal(map[string]any{"handoff_ref": ref, "anchor_sha": strings.Repeat("a", 40), "harness_id": "worker", "lease_epoch": first.Lease.Epoch, "expected_version": first.Version})
|
||||
if err := s.Append(domain.Event{ID: "release", Type: "TaskReleased", TaskID: "t", Version: first.Version + 1, Payload: release, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Lease("t", "worker", time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second, _ := s.Task("t")
|
||||
if second.Lease.Epoch == first.Lease.Epoch {
|
||||
t.Fatal("re-lease reused fencing epoch")
|
||||
}
|
||||
report, err := s.PutArtifact([]byte("report"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
staleComplete, _ := json.Marshal(map[string]any{"report_ref": report, "receipt": map[string]any{"consumed": 1}, "harness_id": "worker", "lease_epoch": first.Lease.Epoch, "expected_version": second.Version})
|
||||
if err := s.Append(domain.Event{ID: "stale-complete", Type: "TaskCompleted", TaskID: "t", Version: second.Version + 1, Payload: staleComplete, Surface: string(authz.System)}); !errors.Is(err, domain.ErrConflict) {
|
||||
t.Fatalf("stale completion = %v, want conflict", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenRebuildsOnlyFromLogAndIgnoresCorruptSnapshot(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
s, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Append(created("create")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "snapshot.json"), []byte(`not json`), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
restarted, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("corrupt disposable snapshot prevented log recovery: %v", err)
|
||||
}
|
||||
if task, ok := restarted.Task("task-1"); !ok || task.State != domain.StateQueued {
|
||||
t.Fatalf("log projection = %#v, present=%v", task, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReplayLegacyLeaseDerivesNonRenewableFence(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
until := time.Now().Add(time.Hour).UnixNano()
|
||||
events := []domain.Event{
|
||||
{SchemaVersion: 2, Seq: 1, ID: "create", Type: "TaskCreated", TaskID: "t", Version: 1, Payload: []byte(`{"source":"s","external_id":"legacy","project":"p"}`), Surface: string(authz.System)},
|
||||
{SchemaVersion: 2, Seq: 2, ID: "lease", Type: "TaskLeased", TaskID: "t", Version: 2, Payload: []byte(`{"harness_id":"worker","until_ns":` + fmt.Sprint(until) + `,"expected_version":1}`), Surface: string(authz.System)},
|
||||
}
|
||||
f, err := os.Create(filepath.Join(dir, "events.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, e := range events {
|
||||
b, _ := json.Marshal(e)
|
||||
if _, err := f.Write(append(b, '\n')); err != nil {
|
||||
_ = f.Close()
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, _ := s.Task("t")
|
||||
if task.Lease == nil || task.Lease.Epoch != "legacy:lease" {
|
||||
t.Fatalf("legacy lease fence = %#v", task.Lease)
|
||||
}
|
||||
if _, err := s.RenewLease("t", "worker", "", task.Version, time.Hour); !errors.Is(err, domain.ErrConflict) {
|
||||
t.Fatalf("legacy lease renewed without derived fence: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlockedTaskProjectsStructuredDiagnosisAndLegacyFallback(t *testing.T) {
|
||||
s, err := Open(t.TempDir())
|
||||
if err != nil {
|
||||
@@ -407,7 +504,8 @@ func TestReleaseTransactionSurvivesReLeaseUntilMatchingPickup(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
anchor := strings.Repeat("a", 40)
|
||||
p, _ := json.Marshal(map[string]any{"handoff_ref": ref, "anchor_sha": anchor, "transaction_id": "tx-1", "expected_version": leased.Version})
|
||||
first, _ := s.Task("task-1")
|
||||
p, _ := json.Marshal(map[string]any{"handoff_ref": ref, "anchor_sha": anchor, "transaction_id": "tx-1", "harness_id": first.Lease.HarnessID, "lease_epoch": first.Lease.Epoch, "expected_version": leased.Version})
|
||||
if err := s.Append(domain.Event{ID: "release", Type: "TaskReleased", TaskID: "task-1", Version: leased.Version + 1, Payload: p, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -418,7 +516,7 @@ func TestReleaseTransactionSurvivesReLeaseUntilMatchingPickup(t *testing.T) {
|
||||
if task.ReleaseTransaction != "tx-1" || task.ReleaseAnchor != anchor || task.HandoffRef != ref {
|
||||
t.Fatalf("re-lease lost transaction: %+v", task)
|
||||
}
|
||||
p, _ = json.Marshal(map[string]any{"transaction_id": "tx-1", "handoff_ref": ref, "anchor_sha": anchor, "harness_id": "successor", "lease_version": task.Version, "expected_version": task.Version})
|
||||
p, _ = json.Marshal(map[string]any{"transaction_id": "tx-1", "handoff_ref": ref, "anchor_sha": anchor, "harness_id": "successor", "lease_epoch": task.Lease.Epoch, "lease_version": task.Version, "expected_version": task.Version})
|
||||
if err := s.Append(domain.Event{ID: "pickup", Type: "TaskPickupValidated", TaskID: "task-1", Version: task.Version + 1, Payload: p, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -422,6 +422,11 @@ func (s Server) action(w http.ResponseWriter, r *http.Request, id, action string
|
||||
http.Error(w, "unknown action", 404)
|
||||
return
|
||||
}
|
||||
if action == "block" {
|
||||
// A browser-created block is an explicit operator decision. Preserve
|
||||
// that fact even if its prose happens to contain a system keyword.
|
||||
body["block_reason"] = string(domain.BlockReasonOperator)
|
||||
}
|
||||
b, _ := json.Marshal(body)
|
||||
e := domain.Event{ID: domain.NewID(), Type: typ, TaskID: id, Version: t.Version + 1, Payload: b, Surface: string(authz.Web)}
|
||||
if err := s.Store.Append(e); err != nil {
|
||||
|
||||
+13
-13
File diff suppressed because one or more lines are too long
+1
-1
File diff suppressed because one or more lines are too long
@@ -1,3 +1,3 @@
|
||||
<script type="module" crossorigin src="/assets/index-D1Up3m4b.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-DUgNKGY0.css">
|
||||
<script type="module" crossorigin src="/assets/index-BXQHTW_a.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-DDZzc9-8.css">
|
||||
<div id="root"></div>
|
||||
|
||||
+574
-356
File diff suppressed because it is too large
Load Diff
Executable
BIN
Binary file not shown.
+4
-1
@@ -7,5 +7,8 @@ RUN npm run build
|
||||
|
||||
FROM nginx:1.27-alpine
|
||||
COPY nginx.conf.template /etc/nginx/templates/default.conf.template
|
||||
COPY --from=build /src/dist/ /usr/share/nginx/html/
|
||||
# vite.config.ts deliberately emits into the Go embed directory so the API
|
||||
# and standalone browser image use the same compiled application. Copy that
|
||||
# fresh build, not the stale source-tree web/dist fallback.
|
||||
COPY --from=build /internal/webui/assets/ /usr/share/nginx/html/
|
||||
EXPOSE 8080
|
||||
|
||||
@@ -16,4 +16,10 @@ describe('UI API client',()=>{
|
||||
await expect(api.upload('report')).resolves.toBe('a'.repeat(64))
|
||||
expect(fetch).toHaveBeenCalledWith('/v1/artifacts',expect.objectContaining({method:'POST',body:'report'}))
|
||||
})
|
||||
it('submits username and password to the browser login endpoint',async()=>{
|
||||
const fetch=vi.fn().mockResolvedValue(new Response(null,{status:204}))
|
||||
vi.stubGlobal('fetch',fetch)
|
||||
await api.login('operator','not stored in the browser')
|
||||
expect(fetch).toHaveBeenCalledWith('/v1/ui/session',expect.objectContaining({method:'POST',body:JSON.stringify({username:'operator',password:'not stored in the browser'})}))
|
||||
})
|
||||
})
|
||||
|
||||
@@ -3,6 +3,6 @@ function sessionExpired(r:Response){if(r.status===401)window.dispatchEvent(new E
|
||||
async function request<T>(path:string, init?:RequestInit):Promise<T>{const r=await fetch(path,{credentials:'same-origin',headers:{'Content-Type':'application/json',...init?.headers},...init});if(!r.ok){sessionExpired(r);throw new Error(await r.text())}return r.json() as Promise<T>}
|
||||
async function text(path:string){const r=await fetch(path);if(!r.ok)throw new Error(await r.text());return r.text()}
|
||||
async function upload(body:string){const r=await fetch('/v1/artifacts',{method:'POST',headers:{'Content-Type':'text/markdown'},body});if(!r.ok)throw new Error(await r.text());return (await r.json() as {ref:string}).ref}
|
||||
async function login(token:string){const r=await fetch('/v1/ui/session',{method:'POST',credentials:'same-origin',headers:{'Content-Type':'application/json'},body:JSON.stringify({token})});if(!r.ok)throw new Error(await r.text())}
|
||||
async function login(username:string,password:string){const r=await fetch('/v1/ui/session',{method:'POST',credentials:'same-origin',headers:{'Content-Type':'application/json'},body:JSON.stringify({username,password})});if(!r.ok)throw new Error(await r.text())}
|
||||
async function logout(){const r=await fetch('/v1/ui/session',{method:'DELETE',credentials:'same-origin'});if(!r.ok)throw new Error(await r.text())}
|
||||
export const api={login,logout,overview:()=>request<Overview>('/v1/ui/overview'),detail:(id:string)=>request<Detail>('/v1/ui/tasks/'+id),artifact:(ref:string)=>text('/v1/ui/artifacts/'+ref),upload,create:(body:unknown)=>request('/v1/ui/tasks',{method:'POST',body:JSON.stringify(body)}),action:(id:string,action:string,body={})=>request<Detail>(`/v1/ui/tasks/${id}/actions/${action}`,{method:'POST',body:JSON.stringify(body)})}
|
||||
|
||||
+8
-4
@@ -13,14 +13,18 @@ const label:Record<TaskState,string>={queued:'Queued',leased:'In session',blocke
|
||||
const blockLabel:Record<BlockReason,string>={lease_failure:'Lease failure',worker_offline:'Worker offline',lease_expired:'Expired lease',approval:'Approval needed',handoff_validation:'Handoff validation',operator_block:'Operator block',system_error:'System error',unknown:'Unknown'}
|
||||
const actionLabel:Record<string,string>={handoff:'Request handoff',release:'Release task',complete:'Complete task',block:'Mark blocked'}
|
||||
const date=(value?:string)=>value?new Date(value).toLocaleString():'—'
|
||||
// Go's zero time can still arrive as "0001-…" despite an omitempty tag on a
|
||||
// value struct. It means this legacy task has no recorded timestamp, not
|
||||
// that it was actually blocked in year one.
|
||||
const recordedTime=(value?:string)=>{if(!value)return undefined;const parsed=new Date(value);return Number.isNaN(parsed.getTime())||parsed.getUTCFullYear()<=1?undefined:value}
|
||||
|
||||
function CommandPalette({close}:{close:()=>void}){const nav=useNavigate(),qc=useQueryClient(),input=useRef<HTMLInputElement>(null),[term,setTerm]=useState('');useEffect(()=>{input.current?.focus()},[]);const choose=(id:string)=>{if(id==='board')nav('/');if(id==='workers')nav('/workers');if(id==='new'){nav('/');window.dispatchEvent(new Event('orchestra:new-task'))}if(id==='refresh')qc.invalidateQueries();close()};const commands=[['board','Go to dispatch board','G B'],['new','Create a new task','N'],['workers','View worker pool','G W'],['refresh','Refresh live data','R']] as const;const matches=commands.filter(c=>c[1].toLowerCase().includes(term.toLowerCase()));return <div className="palette-backdrop" onMouseDown={close}><section className="palette" role="dialog" aria-modal="true" aria-label="Command palette" onMouseDown={e=>e.stopPropagation()}><input ref={input} value={term} onChange={e=>setTerm(e.target.value)} onKeyDown={e=>{if(e.key==='Escape')close();if(e.key==='Enter'&&matches[0])choose(matches[0][0])}} placeholder="Find a command…" aria-label="Find a command"/><div className="palette-list">{matches.map(([id,name,key])=><button key={id} className="palette-command" onClick={()=>choose(id)}><span>{name}</span><kbd>{key}</kbd></button>)}{!matches.length&&<p className="palette-empty">No matching command.</p>}</div><p className="palette-foot"><kbd>↵</kbd> run <kbd>esc</kbd> close</p></section></div>}
|
||||
|
||||
function Shell({children,onLogout}:{children:React.ReactNode;onLogout:()=>void}){const where=useLocation(),nav=useNavigate(),[palette,setPalette]=useState(false),[accountOpen,setAccountOpen]=useState(false);const overview=useQuery({queryKey:['overview'],queryFn:api.overview,refetchInterval:5000});const sessions=overview.data?.sessions?.length??0;useEffect(()=>{const key=(e:KeyboardEvent)=>{const editable=e.target instanceof HTMLInputElement||e.target instanceof HTMLTextAreaElement;if((e.metaKey||e.ctrlKey)&&e.key.toLowerCase()==='k'){e.preventDefault();setPalette(true)}if(e.key==='n'&&!e.metaKey&&!e.ctrlKey&&!editable){nav('/');window.dispatchEvent(new Event('orchestra:new-task'))}if(e.key==='r'&&!e.metaKey&&!e.ctrlKey&&!editable)overview.refetch()};window.addEventListener('keydown',key);return()=>window.removeEventListener('keydown',key)},[nav,overview]);return <div className="app-shell" data-app="orchestra"><aside className="rail"><Link className="mark" to="/" aria-label="Orchestra home"><i className="branch-mark"/><span>OR</span></Link><nav className="rail-nav" aria-label="Primary navigation"><NavLink end className={({isActive})=>'rail-link '+(isActive?'active':'')} to="/"><span>Board</span></NavLink><NavLink className={({isActive})=>'rail-link '+(isActive?'active':'')} to="/workers"><span>Workers</span></NavLink></nav><span className="rail-footer">v0.1</span></aside><header className="topbar"><div className="topbar-title">Orchestra <small>{where.pathname==='/'?'dispatch board':where.pathname==='/workers'?'worker pool':'task record'}</small></div><button className="command" onClick={()=>setPalette(true)} aria-label="Open command palette"><span>Command</span><kbd>⌘ K</kbd></button><div className="readout"><span>SESSIONS <b>{sessions}</b></span><span>SYNC <b>5s</b></span></div><div className="account"><button className="account-button" onClick={()=>setAccountOpen(v=>!v)} aria-expanded={accountOpen}>Signed in <span className="status-dot"/></button>{accountOpen&&<div className="account-menu"><span>Browser session active</span><button onClick={onLogout}>Sign out</button></div>}</div></header>{children}{palette&&<CommandPalette close={()=>setPalette(false)}/>}</div>}
|
||||
|
||||
function taskExplanation(task:Task,overview:Overview){const session=(overview.sessions??[]).find(s=>s.capture?.task_id===task.id);if(task.state==='queued'){const online=(overview.workers??[]).filter(w=>w.online);return online.length?`Waiting for a worker to lease it · ${online.length} worker${online.length===1?'':'s'} online`:'No registered worker is currently reachable'}if(task.state==='leased'){if(session?.pending_approval)return 'Waiting for an operator approval';if(task.lease&&new Date(task.lease.until)<new Date())return 'Lease has expired; waiting for reconciliation';if(session?.blocker)return session.blocker;return session?.capture?'Harness session is active':'Lease is active; live capture is unavailable'}if(task.state==='blocked'){const evidence=task.last_session;const observed=evidence?.captured_at||evidence?.checked_at;const suffix=evidence?` · ${evidence.source||'unknown'} observed ${date(observed)}`:'';return (task.blocker||'Unknown — legacy record has no retained blocker or pane evidence')+suffix}if(task.state==='failed')return 'The task needs review before it can be retried';return 'Terminal task record'}
|
||||
function TaskCard({task,overview}:{task:Task;overview:Overview}){return <Link className="card" to={'/tasks/'+task.id}><b>{task.title||task.id}</b><small><span>{task.project}</span><span className="machine">{task.id.slice(-5)}</span></small><em>{taskExplanation(task,overview)}</em></Link>}
|
||||
function Board({tasks,overview,empty}:{tasks:Task[];overview:Overview;empty:string}){const blocked=tasks.filter(t=>t.state==='blocked').sort((a,b)=>new Date(a.blocked_at||0).getTime()-new Date(b.blocked_at||0).getTime());const groups=new Map<BlockReason,Task[]>;for(const task of blocked){const reason=task.block_reason||'unknown';groups.set(reason,[...(groups.get(reason)||[]),task])}if(!tasks.length)return <section className="empty-panel"><i className="branch-mark"/><h2>No matching work</h2><p>{empty}</p></section>;const boardStates=states.filter(state=>tasks.some(t=>t.state===state));return <><div className="board">{boardStates.map(state=>{const lane=tasks.filter(t=>t.state===state);return <section key={state}><div className="lane-head"><span>{label[state]}</span><span className="lane-count">{lane.length}</span></div>{lane.map(t=><TaskCard task={t} overview={overview} key={t.id}/>)}</section>})}</div>{blocked.length>0&&<section className="blocked-groups" aria-label="Blocked work by diagnosis"><div className="section-title"><div><h2>Needs attention</h2><p>Blocked work is grouped by diagnosis and ordered oldest first.</p></div><span className="machine">{blocked.length} blocked</span></div><div className="blocked-grid">{[...groups.entries()].map(([reason,group])=><section key={reason}><div className="lane-head"><span>{blockLabel[reason]}</span><span className="lane-count">{group.length}</span></div>{group.map(task=><TaskCard task={task} overview={overview} key={task.id}/>)}</section>)}</div></section>}</>}
|
||||
function TaskCard({task,overview,showBlockedAt=false}:{task:Task;overview:Overview;showBlockedAt?:boolean}){const blockedAt=recordedTime(task.blocked_at);return <Link className="card" to={'/tasks/'+task.id}><b>{task.title||task.id}</b><small><span>{task.project}</span><span className="machine">{task.id.slice(-5)}</span></small><em>{taskExplanation(task,overview)}</em>{showBlockedAt&&blockedAt&&<time className="card-age" dateTime={blockedAt}>Blocked since {date(blockedAt)}</time>}</Link>}
|
||||
function Board({tasks,overview,empty}:{tasks:Task[];overview:Overview;empty:string}){const blocked=tasks.filter(t=>t.state==='blocked').sort((a,b)=>new Date(a.blocked_at||0).getTime()-new Date(b.blocked_at||0).getTime());const groups=new Map<BlockReason,Task[]>;for(const task of blocked){const reason=task.block_reason||'unknown';groups.set(reason,[...(groups.get(reason)||[]),task])}if(!tasks.length)return <section className="empty-panel"><i className="branch-mark"/><h2>No matching work</h2><p>{empty}</p></section>;const boardStates=states.filter(state=>tasks.some(t=>t.state===state));return <><div className="board">{boardStates.map(state=>{const lane=tasks.filter(t=>t.state===state);return <section key={state}><div className="lane-head"><span>{label[state]}</span><span className="lane-count">{lane.length}</span></div>{lane.map(t=><TaskCard task={t} overview={overview} key={t.id}/>)}</section>})}</div>{blocked.length>0&&<section className="blocked-groups" aria-label="Blocked work by diagnosis"><div className="section-title"><div><h2>Needs attention</h2><p>Blocked work is grouped by diagnosis and ordered oldest first.</p></div><span className="machine">{blocked.length} blocked</span></div><div className="blocked-grid">{[...groups.entries()].map(([reason,group])=><section key={reason}><div className="lane-head"><span>{blockLabel[reason]}</span><span className="lane-count">{group.length}</span></div>{group.map(task=><TaskCard task={task} overview={overview} showBlockedAt key={task.id}/>)}</section>)}</div></section>}</>}
|
||||
|
||||
function OverviewPage(){const q=useQuery({queryKey:['overview'],queryFn:api.overview,refetchInterval:5000}),[createOpen,setCreateOpen]=useState(false),[view,setView]=useState<'active'|'history'|'all'>('active'),[project,setProject]=useState(''),[search,setSearch]=useState('');useEffect(()=>{const open=()=>setCreateOpen(true);window.addEventListener('orchestra:new-task',open);return()=>window.removeEventListener('orchestra:new-task',open)},[]);if(q.isLoading)return <main className="page loading"><p className="eyebrow">Dispatch board</p><p>Fetching queue state from the coordinator…</p></main>;if(q.error)return <main className="page error" role="alert">Queue unavailable: {String(q.error)}</main>;const d=q.data!,sessions=d.sessions??[],active=d.tasks.filter(t=>t.state==='leased').length,attention=d.tasks.filter(t=>t.state==='blocked'||t.state==='failed').length,approvals=new Set(sessions.filter(s=>s.pending_approval).map(s=>s.capture?.task_id)),projects=[...new Set(d.tasks.map(t=>t.project).filter(Boolean))].sort(),term=search.trim().toLowerCase(),filtered=d.tasks.filter(t=>(!project||t.project===project)&&(!term||[t.id,t.title,t.description,t.project,t.blocker].filter(Boolean).join(' ').toLowerCase().includes(term))).filter(t=>view==='all'||view==='active'?activeStates.includes(t.state):!activeStates.includes(t.state)),history=d.tasks.filter(t=>!activeStates.includes(t.state)).length,empty=view==='active'?'There is no live work right now. Completed and failed records are available in History.':view==='history'?'No completed or failed records match these filters.':'No task records match these filters.';return <main className="page"><div className="page-heading"><div><p className="eyebrow">Agent dispatch</p><h1>Keep the work moving.</h1></div><div className="heading-actions"><p>Live task state across every connected harness. The board refreshes every five seconds.</p><button onClick={()=>setCreateOpen(true)}>New task <kbd>N</kbd></button></div></div><div className="status-strip" aria-label="Queue summary"><div><span>Tasks</span><b>{d.tasks.length}</b></div><div><span>In session</span><b>{active}</b></div><div className={attention?'attention':''}><span>Needs attention</span><b>{attention}</b></div><div className={approvals.size?'attention':''}><span>Awaiting approval</span><b>{approvals.size}</b></div></div><div className="section-title"><div><h2>Task flow</h2><p>{view==='active'?'Live, waiting, and needs-attention work.':view==='history'?'Completed and failed task records.':'All matching task records.'}</p></div><span className="machine" aria-live="polite">SYNCED {new Date(d.updated_at).toLocaleTimeString()}</span></div><div className="board-filters" aria-label="Task board filters"><div className="view-tabs" role="group" aria-label="Task view"><button className={view==='active'?'selected':''} onClick={()=>setView('active')}>Live work</button><button className={view==='history'?'selected':''} onClick={()=>setView('history')}>History {history>0&&<span>{history}</span>}</button><button className={view==='all'?'selected':''} onClick={()=>setView('all')}>All</button></div><label className="sr-only" htmlFor="task-search">Search tasks</label><input id="task-search" value={search} onChange={e=>setSearch(e.target.value)} placeholder="Search tasks"/><label className="sr-only" htmlFor="project-filter">Filter by project</label><select id="project-filter" value={project} onChange={e=>setProject(e.target.value)}><option value="">All projects</option>{projects.map(p=><option value={p} key={p}>{p}</option>)}</select></div><Board tasks={filtered} overview={d} empty={empty}/>{createOpen&&<Create overview={d} close={()=>setCreateOpen(false)}/>}</main>}
|
||||
|
||||
@@ -40,7 +44,7 @@ function taskDetailExplanation(d:Detail){if(d.session?.pending_approval)return '
|
||||
|
||||
function Workers(){const q=useQuery({queryKey:['overview'],queryFn:api.overview,refetchInterval:5000});if(q.isLoading)return <main className="page loading">Fetching worker heartbeats…</main>;if(q.error)return <main className="page error" role="alert">Worker pool unavailable: {String(q.error)}</main>;const d=q.data!,workers=d.workers;const herdr=(status?:string)=>status==='reachable'?'local herdr reachable':status==='unreachable'?'local herdr unreachable':'local herdr not yet checked';return <main className="page"><Link className="back" to="/">← Dispatch board</Link><div className="page-heading"><div><p className="eyebrow">Worker pool</p><h1>Available capacity.</h1></div><p>{workers.filter(w=>w.online).length} of {workers.length} registered workers are reachable right now.</p></div>{workers.length?<><p className="worker-note">Heartbeat proves the worker can reach Orchestra. Local herdr status is separately reported by that worker; it is never inferred from legacy coordinator TCP probes.</p><table><thead><tr><th>Worker</th><th>Heartbeat</th><th>Local herdr</th><th>Active work</th><th>Last error</th><th>Last heartbeat</th></tr></thead><tbody>{workers.map(w=><tr key={w.id}><td>{w.id}</td><td className={w.online?'online':'offline'}><span className="status-dot"/> {w.online?'reachable':'overdue'}</td><td className={w.health?.herdr_status==='reachable'?'online':w.health?.herdr_status==='unreachable'?'offline':''}>{herdr(w.health?.herdr_status)}<small>{date(w.health?.checked_at)}</small></td><td>{w.health?.active_task_id?<><span>{w.health.active_task_id}</span><small>{w.health.active_pane_id||'pane unknown'}</small></>:'idle'}</td><td>{w.health?.last_error?<><span>{w.health.last_error}</span><small>{date(w.health.error_at)}</small></>:'—'}</td><td>{date(w.last_seen)}</td></tr>)}</tbody></table></>:<section className="empty-panel"><i className="branch-mark"/><h2>No workers registered</h2><p>Connect a worker to begin leasing queued tasks.</p></section>}</main>}
|
||||
function Artifact(){const {ref=''}=useParams();const q=useQuery({queryKey:['artifact',ref],queryFn:()=>api.artifact(ref)});return <main className="page"><Link className="back" to="/">← Dispatch board</Link><div className="page-heading"><div><p className="eyebrow">Evidence artifact</p><h1>Recorded output.</h1></div><p className="machine">{ref}</p></div>{q.isLoading?<p className="loading">Retrieving artifact…</p>:q.error?<p className="error" role="alert">{String(q.error)}</p>:<pre>{q.data}</pre>}</main>}
|
||||
function Login({onAuthenticated,message}:{onAuthenticated:()=>void;message?:string}){const [token,setToken]=useState(''),[error,setError]=useState(''),[pending,setPending]=useState(false);const submit=async(e:React.FormEvent)=>{e.preventDefault();setError('');setPending(true);try{await api.login(token);setToken('');onAuthenticated()}catch(err){setError(String(err))}finally{setPending(false)}};return <main className="login-page"><section className="login-card"><p className="eyebrow">Orchestra control plane</p><h1>Sign in</h1><p>Enter the Web token to open a secure browser session.</p>{message&&<p className="session-message" role="status">{message}</p>}<form onSubmit={submit}><label>Web token<input type="password" autoComplete="current-password" autoFocus value={token} onChange={e=>setToken(e.target.value)} required/></label><button disabled={pending}>{pending?'Signing in…':'Sign in'}</button>{error&&<span className="error" role="alert">{error}</span>}</form></section></main>}
|
||||
function Login({onAuthenticated,message}:{onAuthenticated:()=>void;message?:string}){const input=useRef<HTMLInputElement>(null),[username,setUsername]=useState(''),[password,setPassword]=useState(''),[error,setError]=useState(''),[pending,setPending]=useState(false),[visible,setVisible]=useState(false),[capsLock,setCapsLock]=useState(false);useEffect(()=>input.current?.focus(),[]);const submit=async(e:React.FormEvent)=>{e.preventDefault();if(!username||!password){setError('Enter your username and password to continue.');(!username?input.current:document.getElementById('web-password'))?.focus();return}setError('');setPending(true);try{await api.login(username,password);setPassword('');onAuthenticated()}catch{setError('That username or password was not accepted. Check both and try again.')}finally{setPending(false)}};return <main className="login-page"><div className="login-orbit login-orbit-one"/><div className="login-orbit login-orbit-two"/><section className="login-card" aria-labelledby="login-title"><header className="login-brand"><i className="branch-mark" aria-hidden="true"/><div><span>ORCHESTRA</span><small>CONTROL PLANE</small></div></header><div className="login-heading"><p className="eyebrow">Operator access</p><h1 id="login-title">Welcome back.</h1><p>Sign in to open this browser’s operator session.</p></div>{message&&<p className="session-message" role="status">{message}</p>}<form onSubmit={submit} noValidate><label htmlFor="web-username">Username</label><input ref={input} id="web-username" autoComplete="username" autoCapitalize="none" autoCorrect="off" spellCheck={false} value={username} onChange={e=>{setUsername(e.target.value);if(error)setError('')}} aria-invalid={!!error} placeholder="Operator username"/><label htmlFor="web-password">Password</label><div className="token-field"><input id="web-password" type={visible?'text':'password'} autoComplete="current-password" autoCapitalize="none" autoCorrect="off" spellCheck={false} value={password} onChange={e=>{setPassword(e.target.value);if(error)setError('')}} onKeyUp={e=>setCapsLock(e.getModifierState('CapsLock'))} onKeyDown={e=>setCapsLock(e.getModifierState('CapsLock'))} aria-invalid={!!error} aria-describedby="login-help login-error" placeholder="Password"/><button type="button" className="reveal-token" onClick={()=>setVisible(v=>!v)} aria-label={visible?'Hide password':'Show password'} aria-pressed={visible}>{visible?'Hide':'Show'}</button></div><div className="token-status"><span id="login-help">Your password is verified against a bcrypt hash and never stored in this browser.</span>{capsLock&&<span className="caps-lock">Caps Lock is on</span>}</div>{error&&<p id="login-error" className="login-error" role="alert">{error}</p>}<button className="login-submit" disabled={pending||!username||!password}>{pending?<><span className="button-spinner"/>Opening session…</>:'Sign in'}</button></form><footer className="login-security"><span className="login-lock" aria-hidden="true">⌁</span><p>Successful sign-in creates an HttpOnly, same-site session cookie. Sessions expire after 12 hours.</p></footer></section></main>}
|
||||
function RoutesApp({onLogout}:{onLogout:()=>void}){return <Shell onLogout={onLogout}><Routes><Route path="/" element={<OverviewPage/>}/><Route path="/tasks/:taskID" element={<TaskDetail/>}/><Route path="/workers" element={<Workers/>}/><Route path="/artifacts/:ref" element={<Artifact/>}/></Routes></Shell>}
|
||||
function App(){const [ready,setReady]=useState(false),[checking,setChecking]=useState(true),[message,setMessage]=useState('');useEffect(()=>{const unauth=()=>{client.clear();setMessage('Your browser session expired. Sign in again to continue.');setReady(false);setChecking(false)};window.addEventListener('orchestra:unauthorized',unauth);api.overview().then(()=>setReady(true)).catch(()=>setReady(false)).finally(()=>setChecking(false));return()=>window.removeEventListener('orchestra:unauthorized',unauth)},[]);const logout=async()=>{await api.logout();client.clear();setMessage('You have signed out.');setReady(false)};if(checking)return <main className="login-page">Checking session…</main>;return ready?<RoutesApp onLogout={logout}/>:<Login message={message} onAuthenticated={()=>{client.clear();setMessage('');setReady(true)}}/>}
|
||||
function App(){const [ready,setReady]=useState(false),[checking,setChecking]=useState(true),[message,setMessage]=useState('');useEffect(()=>{const unauth=()=>{client.clear();setMessage('Your browser session expired. Sign in again to continue.');setReady(false);setChecking(false)};window.addEventListener('orchestra:unauthorized',unauth);api.overview().then(()=>setReady(true)).catch(()=>setReady(false)).finally(()=>setChecking(false));return()=>window.removeEventListener('orchestra:unauthorized',unauth)},[]);const logout=async()=>{await api.logout();client.clear();setMessage('You have signed out.');setReady(false)};if(checking)return <main className="login-page"><section className="session-check" aria-live="polite"><i className="branch-mark" aria-hidden="true"/><span className="button-spinner"/>Checking your session…</section></main>;return ready?<RoutesApp onLogout={logout}/>:<Login message={message} onAuthenticated={()=>{client.clear();setMessage('');setReady(true)}}/>}
|
||||
createRoot(document.getElementById('root')!).render(<React.StrictMode><QueryClientProvider client={client}><BrowserRouter><App/></BrowserRouter></QueryClientProvider></React.StrictMode>)
|
||||
|
||||
+2
-2
@@ -7,13 +7,13 @@
|
||||
@media(max-width:760px){.status-strip{grid-template-columns:1fr 1fr;margin-bottom:24px}.status-strip div:nth-child(2){border-right:0}.status-strip div:nth-child(-n+2){border-bottom:1px solid var(--line)}.palette-backdrop{padding-top:9vh}.palette{max-height:82vh}.palette-command{padding:13px 11px}}
|
||||
|
||||
.heading-actions{display:flex;align-items:end;gap:16px}.heading-actions button{white-space:nowrap}.modal-backdrop{position:fixed;inset:0;z-index:15;display:grid;place-items:center;padding:20px;background:rgba(8,7,5,.72)}.create.modal{width:min(680px,100%);max-height:calc(100vh - 40px);overflow:auto}.modal-heading{display:flex;align-items:start;justify-content:space-between;gap:12px}.modal-heading h2{margin:0}.icon-button{display:grid;width:30px;height:30px;place-items:center;padding:0;border-color:var(--line-hi);color:var(--text-mid);background:transparent;font-size:21px;font-weight:400}.create form{display:grid;gap:9px}.create form label{display:grid;gap:6px;color:var(--text-mid);font-size:12px}.create form input,.create form textarea{margin:0}.modal-actions{display:flex;justify-content:flex-end;gap:8px;margin-top:4px}.quiet-button{border-color:var(--line-hi);color:var(--text-mid);background:transparent}
|
||||
.login-page{display:grid;min-height:100vh;place-items:center;padding:20px;color:var(--text-hi)}.login-card{width:min(420px,100%);padding:28px;background:var(--bg-1);border:1px solid var(--line-hi);border-radius:var(--r-lg);box-shadow:var(--shadow-soft)}.login-card h1{margin:0 0 8px;font-size:31px;letter-spacing:-.04em}.login-card>p:not(.eyebrow){margin:0 0 22px;color:var(--text-mid);line-height:1.5}.login-card form{display:grid;gap:12px}.login-card label{display:grid;gap:6px;color:var(--text-mid);font-size:12px}
|
||||
.login-page{position:relative;display:grid;min-height:100vh;place-items:center;overflow:hidden;padding:24px;color:var(--text-hi);background:radial-gradient(ellipse 70% 55% at 50% 0%,rgba(124,150,120,.16),transparent 70%),var(--bg-0)}.login-page:before{position:absolute;inset:0;background:linear-gradient(115deg,transparent 0 47%,rgba(244,234,220,.025) 47% 47.1%,transparent 47.1% 100%);content:'';pointer-events:none}.login-orbit{position:absolute;border:1px solid rgba(158,183,153,.12);border-radius:50%;pointer-events:none}.login-orbit-one{width:min(85vw,940px);height:min(85vw,940px);transform:translate(38%,-40%)}.login-orbit-two{width:min(62vw,650px);height:min(62vw,650px);transform:translate(-52%,53%)}.login-card{position:relative;width:min(460px,100%);padding:30px;background:linear-gradient(145deg,rgba(44,38,29,.96),rgba(27,23,18,.98));border:1px solid var(--line-hi);border-radius:20px;box-shadow:0 26px 80px rgba(0,0,0,.42),var(--shadow-soft)}.login-brand{display:flex;align-items:center;gap:10px;margin-bottom:35px;color:var(--accent-hi)}.login-brand .branch-mark{height:25px}.login-brand div{display:grid;gap:2px;font:600 14px var(--mono);letter-spacing:-.08em}.login-brand small{color:var(--text-lo);font:9px var(--sans);letter-spacing:.15em}.login-heading{margin-bottom:25px}.login-heading .eyebrow{margin-bottom:8px}.login-card h1{margin:0 0 8px;font-size:36px;line-height:1;letter-spacing:-.055em}.login-heading>p:not(.eyebrow){max-width:360px;margin:0;color:var(--text-mid);font-size:14px;line-height:1.55}.login-card form{display:grid;gap:10px}.login-card form>label{color:var(--text-mid);font-size:12px;font-weight:600}.token-field{position:relative}.token-field input{padding-right:68px;border-color:var(--line-hi);font-family:var(--mono);font-size:13px}.token-field input[aria-invalid="true"]{border-color:#d5a092;box-shadow:0 0 0 3px rgba(213,160,146,.1)}.reveal-token{position:absolute;top:50%;right:5px;transform:translateY(-50%);padding:6px 8px;border:0;color:var(--accent-hi);background:transparent;font-size:11px}.reveal-token:hover{background:var(--accent-dim)}.token-status{display:flex;justify-content:space-between;gap:10px;color:var(--text-lo);font-size:11px;line-height:1.4}.caps-lock{color:#d5a092}.login-error{margin:2px 0;padding:9px 10px;border:1px solid rgba(213,160,146,.5);border-radius:var(--r-sm);color:#e4b0a2;background:rgba(213,160,146,.09);font-size:12px;line-height:1.4}.login-submit{display:flex;min-height:42px;align-items:center;justify-content:center;gap:8px;margin-top:8px;font-size:13px}.button-spinner{display:inline-block;width:13px;height:13px;border:2px solid currentColor;border-right-color:transparent;border-radius:50%;animation:spin .75s linear infinite}.login-security{display:flex;gap:10px;margin-top:24px;padding-top:17px;border-top:1px solid var(--line);color:var(--text-lo)}.login-security p{margin:0;font-size:11px;line-height:1.5}.login-lock{color:var(--accent-hi);font:18px var(--mono)}.session-check{position:relative;display:flex;align-items:center;gap:11px;padding:16px 18px;border:1px solid var(--line-hi);border-radius:var(--r-md);background:var(--bg-1);color:var(--text-mid);font-size:13px;box-shadow:var(--shadow-soft)}.session-check .branch-mark{height:21px;color:var(--accent-hi)}@keyframes spin{to{transform:rotate(360deg)}}
|
||||
@media(max-width:760px){.heading-actions{width:100%;align-items:stretch;flex-direction:column}.heading-actions button{align-self:flex-start}.modal-backdrop{padding:12px}.create.modal{max-height:calc(100vh - 24px)}}
|
||||
.blocked-groups{margin-top:32px}.blocked-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(230px,1fr));gap:10px}.blocked-grid>section{min-height:120px;padding:12px;background:var(--bg-1);border:1px solid rgba(213,160,146,.34);border-radius:var(--r-md)}
|
||||
|
||||
/* Operator comfort pass: make live state scannable before the raw data. */
|
||||
.account{position:relative}.account-button{display:flex;align-items:center;gap:7px;padding:7px 9px;border-color:var(--line);color:var(--text-mid);background:transparent;font-size:11px;font-weight:500}.account-menu{position:absolute;top:calc(100% + 8px);right:0;z-index:10;display:grid;min-width:190px;gap:8px;padding:10px;border:1px solid var(--line-hi);border-radius:var(--r-sm);background:var(--bg-2);box-shadow:var(--shadow-soft);color:var(--text-lo);font-size:11px}.account-menu button{padding:7px 9px;text-align:left}.status-dot{display:inline-block;width:7px;height:7px;border-radius:50%;background:var(--accent-hi);box-shadow:0 0 0 3px var(--accent-dim)}
|
||||
.section-title>div{display:grid;gap:3px}.section-title p,.section-copy,.worker-note{margin:0;color:var(--text-lo);font-size:12px;line-height:1.5}.card em{display:block;overflow:hidden;margin-top:9px;color:var(--text-lo);font-size:11px;font-style:normal;line-height:1.35;text-overflow:ellipsis;white-space:nowrap}.card:hover em{color:var(--text-mid)}.eligibility{display:grid;gap:3px;padding:10px 12px;border:1px solid var(--accent-line);border-radius:var(--r-sm);background:var(--accent-dim);color:var(--text-mid);font-size:12px}.eligibility b{color:var(--accent-hi);font-size:12px}.eligibility.warning{border-color:rgba(213,160,146,.45);background:rgba(213,160,146,.09)}.eligibility.warning b{color:#d5a092}
|
||||
.section-title>div{display:grid;gap:3px}.section-title p,.section-copy,.worker-note{margin:0;color:var(--text-lo);font-size:12px;line-height:1.5}.card em{display:block;overflow:hidden;margin-top:9px;color:var(--text-lo);font-size:11px;font-style:normal;line-height:1.35;text-overflow:ellipsis;white-space:nowrap}.card:hover em{color:var(--text-mid)}.card-age{display:block;margin-top:8px;padding-top:8px;border-top:1px solid var(--line);color:#d5a092;font:10px var(--mono);line-height:1.3}.eligibility{display:grid;gap:3px;padding:10px 12px;border:1px solid var(--accent-line);border-radius:var(--r-sm);background:var(--accent-dim);color:var(--text-mid);font-size:12px}.eligibility b{color:var(--accent-hi);font-size:12px}.eligibility.warning{border-color:rgba(213,160,146,.45);background:rgba(213,160,146,.09)}.eligibility.warning b{color:#d5a092}
|
||||
.board-filters{display:flex;align-items:center;gap:8px;margin:-2px 0 16px}.view-tabs{display:flex;gap:4px;margin-right:auto;padding:3px;border:1px solid var(--line);border-radius:var(--r-sm);background:var(--bg-1)}.view-tabs button{padding:6px 9px;border-color:transparent;color:var(--text-lo);background:transparent;font-size:11px;font-weight:500}.view-tabs button.selected{border-color:var(--accent-line);color:var(--accent-hi);background:var(--accent-dim)}.view-tabs span{margin-left:4px;color:var(--text-machine);font:10px var(--mono)}.board-filters input,.board-filters select{width:auto;min-width:150px;padding:8px 10px;border:1px solid var(--line-hi);border-radius:var(--r-sm);outline:none;color:var(--text-mid);background:var(--bg-1);font-size:12px}.board-filters select{cursor:pointer}.board-filters input:focus,.board-filters select:focus{border-color:var(--accent)}
|
||||
.task-banner{display:flex;align-items:center;justify-content:space-between;gap:20px;margin:-8px 0 20px;padding:13px 16px;border:1px solid var(--accent-line);border-radius:var(--r-md);background:var(--accent-dim)}.task-banner>div{display:flex;align-items:center;gap:12px}.task-banner .task-state{margin:0;color:var(--accent-hi);font-weight:600}.task-banner p{margin:0;color:var(--text-mid);font-size:13px;line-height:1.45}.task-banner>span{color:var(--text-lo);font:11px var(--mono);white-space:nowrap}.task-banner>span b{color:var(--text-machine);font-weight:500}.task-banner.state-blocked,.task-banner.state-failed{border-color:rgba(213,160,146,.42);background:rgba(213,160,146,.08)}
|
||||
.diagnosis{display:grid;gap:17px;margin:-8px 0 20px;padding:20px;border:1px solid var(--accent-line);border-radius:var(--r-md);background:var(--accent-dim)}.diagnosis.state-blocked,.diagnosis.state-failed{border-color:rgba(213,160,146,.42);background:rgba(213,160,146,.08)}.diagnosis-heading{display:flex;align-items:start;justify-content:space-between;gap:20px}.diagnosis .eyebrow{margin-bottom:6px}.diagnosis h2{margin:0 0 7px;font-size:22px;letter-spacing:-.03em}.diagnosis-heading p:not(.eyebrow){max-width:780px;margin:0;color:var(--text-mid);font-size:13px;line-height:1.5}.diagnosis .task-state{flex:none;margin:0;color:var(--accent-hi);font-weight:600}.diagnosis-evidence{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:10px;margin:0}.diagnosis-evidence div{min-width:0;padding:10px 11px;border:1px solid var(--line);border-radius:var(--r-sm);background:rgba(20,17,13,.28)}.diagnosis-evidence dt{margin-bottom:5px;color:var(--text-lo);font-size:10px;letter-spacing:.07em;text-transform:uppercase}.diagnosis-evidence dd{overflow:hidden;margin:0;color:var(--text-machine);font-size:12px;line-height:1.35;text-overflow:ellipsis;white-space:nowrap}.next-action{display:flex;align-items:center;justify-content:space-between;gap:18px;padding:13px 14px;border-left:3px solid var(--accent);background:rgba(20,17,13,.35)}.next-action>div{display:grid;gap:3px}.next-action span{color:var(--accent-hi);font-size:10px;font-weight:600;letter-spacing:.1em;text-transform:uppercase}.next-action b{font-size:14px}.next-action p{margin:0;color:var(--text-mid);font-size:12px;line-height:1.45}.next-action small{flex:none;color:var(--text-lo);font:11px var(--mono)}
|
||||
|
||||
Reference in New Issue
Block a user