F41, live on run 5: the review agent produced nothing after 02:12 and the
02:46 renewal was granted anyway. The progress digest covered Claude Code's
status footer, and one of its fields ticked inside the window. Reproduced the
worker's stored progress_sha byte for byte from the live pane, so the branch
taken was progress != ProgressSHA, not IsBusy and not an empty baseline.
PaneProgress now cuts from the editor's lower rule and trims the spinner
summary and version notice above it. AgentStatus still reads the raw capture,
so the busy markers living in the footer are unaffected.
Lease TTL moves to 5 minutes, from domain.LeaseTTL, with renewal at half of
it. Reclaiming a stalled pane happens only at expiry, and 30 minutes per
window made run 5's stall unbounded in practice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
A task that reached the router's MaxAttempts was permanently terminal.
TaskReleased only ever increments Attempt, TaskCorrected could not touch it,
and no HTTP route emitted a correction at all. The only way to work an
exhausted issue again was to invent a second task for it, which defeats
(source, external_id) dedupe and abandons the task's own history.
POST /v1/tasks/{id}/retry, full-control surfaces only. It requires the task
to be failed, unleased, and failed with reason retry_limit: restoring a retry
budget is not an answer to a failure that was not the budget running out. The
effect is one TaskCorrected naming that failure, setting state queued and
attempt 0 and clearing next_retry_at, failure_class and last_error. Task id,
source pair, goal, acceptance, decisions, work phase and artifact refs all
stay, and the original failure events stay in the log.
operation_id is required and makes the call idempotent, so a repeated request
cannot reset an attempt that has since started running.
This is RetryTask, not a generic correction endpoint: arbitrary task mutation
over HTTP is a different and much larger authority. It also does not address
F9, which is an operator releasing a lease someone else owns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Burn-in run 2 ingested its task and then sat queued forever. Every herdr in
the live config declares quota_limit_5h and quota_limit_weekly, the event log
holds zero QuotaReported events, and QuotaSince reported an empty window as
unknown. QuotaAvailability fails closed on unknown, so no harness could ever
be leased, and the only producer of a receipt is a completed lease.
The event log is Orchestra's whole accounting source, so a window holding no
receipts is observable zero consumption. QuotaSince now reports known for an
empty window and for a harness that has never reported. A receipt that
declares its own consumption unknown still fails closed.
The refusal also lied about its cause. federatedAvailability collapsed a base
gate refusal into the federation health string, so router health said "stale
heartbeat or unhealthy local backend" while the heartbeat was one second old.
Availability gates now name themselves through an optional
ReasonedAvailability contract: quota refusals say whether usage is unknown or
the window is exhausted and by how much, and worker refusals distinguish an
unregistered worker, a never-probed backend, a stale heartbeat, a stale
health check, and an unreachable backend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A task in retry backoff was filtered out before the candidate loop, so it
recorded no rejection at all: queued, apparently assignable, and silent. That is
the exact shape that made F5 take a live session to diagnose. It now reports
"retry backoff until <time>".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
F7, security. An unset surface token makes the middleware skip its check, so a
full-control surface with no credential is an open control plane rather than a
closed one. With ORCHESTRA_TUI_TOKEN unset, any LAN caller could lease, release,
complete or block any task by declaring one header, which is how this session's
manual leases were issued. authz.RequireCredentials now refuses startup instead
of logging. Web is exempt: Sessions makes its login mandatory.
F5, lifecycle. router.go's silent `continue` was the first bug, not the
predicate behind it. Every eligibility gate now records a router.Rejection with
task, herdr and reason, exposed at GET /v1/router/health, reset per pass. No
gate was weakened: a direct Store.Lease succeeding proves the lease path, not
that eligibility should have selected that worker.
F8, correctness. Reconcile iterated every configured source for every task, so a
task's external id was looked up in whatever repository each source pointed at.
Once two repositories share an issue number, an unrelated human comment becomes
an authoritative decision for the wrong task. Reconciliation is now bound to
task.Source, the provider:project identity the ingest stamped, and a source that
cannot prove it owns the task is skipped. A task with no matching source
reconciles to nothing and still launches, because nothing to import is not a
failure to read.
The integration fixture ingested from "jsonl" while reconciling from "gitea",
which is exactly the shape F8 makes impossible; it now ingests from the source
it reconciles.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All five defects filed while implementing B18, plus the router flake that
predated them. None of this has run on the deployed instance: the service
is stopped and /usr/local/bin/orchestra predates every change here.
B20 is the one that could silently defeat approvals. The capture revision
was UnixNano, so it changed on every read and said nothing about whether
the pane had changed; it is now an FNV-1a hash of the pane text, changing
iff the text does. The worse half was precedence: capture() preferred the
coordinator over a published worker capture, handing Queue a timestamp the
owning worker's staleness check could never match, so every federated
approval resolved "stale" and the keystroke never happened. Worker captures
now win — their existence means a registered worker owns that pane — and
capturePane follows the same precedence via Capture.Source rather than
guessing.
B19 was filed as "federated approvals emit no event", which overstated it:
the resolution half already existed, and correctly fires only on an
acknowledged worker report. The missing half was the request. Server.action
now appends ApprovalRequested at queue time, subject_ref set to the command
ID the later resolution carries. If that append fails the queued command is
resolved "rejected" — a keystroke that left no audit trail must not run.
B21 bounds the command list: resolved commands prune after 30 minutes on
both Queue and Commands, pending ones never at any age, since dropping one
would discard an operator decision. The persistence half stays open and is
recorded as such — captures and commands are still in-memory only.
S12 splits ORCHESTRA_NTFY_TOKEN, which was both the secret handed to the
ntfy server and a valid inbound credential for the ntfy surface; the latter
is now ORCHESTRA_NTFY_SURFACE_TOKEN. Breaking: a deployment relying on the
old dual use has no inbound gate until it sets the new variable. S13
deletes the dead auth() copy of the authorization policy.
The router flake was in the test, not in assignment. Store.Tasks() ranges a
map, and the assertion indexed two separate Tasks() calls, failing whenever
the orderings disagreed; instrumenting it showed a valid TaskLeased and a
genuinely leased task on every "failing" run. It now snapshots once and
asserts that exactly one task is leased, and passes at -count=60.
AUDIT.md records what is still not done: the deployed env and binary, the
live re-verification B13-B17 has always lacked, and two operational faults
found in the journal that block it — all six herdrs are refusing
connections, and ntfy delivery is failing 403 on every send.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01535A3Y8RtkAi8wYuWhtkEd
Router.attempts (checked against RetryPolicy.MaxAttempts) advanced on
every TaskReleased and again on every subsequent lease. Rotation is
TaskReleased carrying a valid handoff_ref (spec §5.3: "rotation =
intra-task lease transfer"), not a failure — so a task healthy enough to
rotate twice hit the default MaxAttempts=3 and was killed by the retry
limit meant for genuine failures (expiry, crash).
HandleEvent now only advances attempts (and applies retry backoff) for a
TaskReleased whose payload lacks a handoff_ref. AssignPending no longer
increments attempts at lease time at all — that was double-counting
against the same failure that a subsequent non-rotation release already
counts.
TestRotationDoesNotCountAgainstRetryLimit drives a task through 5
rotate-and-release cycles with MaxAttempts=3 and asserts it never reaches
TaskFailed.
AUDIT.md B4.
Pre-existing uncommitted work found at session start: rotation now emits
anchor_sha on TaskReleased (previously silently dropped by store.Append
validation), multi-repo Gitea provider support, per-project git worktree
roots, and associated test coverage. Committing as a checkpoint before
starting remediation work tracked in AUDIT.md.
kami
kami