herdr hands a freshly created pane/agent back before it's actually ready,
and rejects the very next call with a range of different transient errors
("not an available shell", "not an active named agent", "target ... not
found") depending on timing. String-matching each wording as it turned up
live proved unwinnable across three live redeploy-and-test rounds, so
StartAgent and Prompt now retry any error for up to 15s (bounded by
wall-clock time, not attempt count) rather than pattern-matching herdr's
error text.
Confirmed live against workpc: a fresh lease (wD:p1) now reaches a real
attached claude session instead of failing before the agent starts.
Live testing also exposed a second, separate defect (B13, documented in
AUDIT.md, not fixed here): agent.start can return success while never
actually starting an agent when two leases land close together, with no
error for a retry to catch. Left three test panes on workpc untouched
(wD:p1, wE:p1, wF:p1) pending manual cleanup, per the standing rule against
destructive herdr calls without asking first.
Also folds in the already-flattened AUDIT.md/progress.md merge that was
staged ahead of this session's changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
The previous function_call/function_call_output shape was never verified
and doesn't exist in any real Codex rollout. Confirmed the real shape
against this machine's own ~/.codex/sessions files: file edits arrive as
event_msg/patch_apply_end (changes+success, no pairing needed), and shell
commands arrive as a freeform custom_tool_call named "exec" whose input is
a JS snippet embedding cmd:"..." rather than a flat arguments object, with
failure signaled by a literal "Script error:" prefix in the output text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Closes the last two S11 triggers. internal/herdr/activity.go normalizes
tool/function calls per harness (ClaudeActivity verified against the
existing transcript format, CodexActivity best-effort/unverified,
OpenCodeActivity refuses — no confirmed per-tool-call source exists) and
implements the three thrash rules plus a narrow milestone check
(successful git commit as the last call).
CLIAdapter.RequestHandoffReason asks the agent to write a handoff with
meta.reason set, same "ask, don't invent" pattern as the existing handoff/
report requests. rotate() and TurnDecision generalize the manual-bypass
shortcut to manual/milestone/thrash and request (never directly release)
on a detected trigger.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Implements §3.1's invariant that a wrong event is never edited, only
compensated for by a new appended event. TaskCorrected references the
event it repairs and can change state and/or amend-style fields;
Store.Append verifies the referenced event actually exists on the task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
rotate() and TurnDecision now check the agent's own handoff for
meta.reason=="manual" before evaluating occupancy/turn-boundary — per
spec §5.3, that reason is itself the boundary signal ("a coherent unit
finished and the next is independent"), so it bypasses both checks and
releases immediately. Extracted the shared release-and-anchor-certify
tail into Coordinator.finishRelease so the manual path gets the same
anchor safety guarantee as the threshold path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Coordinator.Soft (default 0.55, ORCHESTRA_OCCUPANCY_SOFT) makes rotate()
and TurnDecision request a handoff advisory-only once occupancy crosses
the soft threshold, well before Hard forces a release/rotation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Register() previously trusted a self-declared id and self-chosen token
from any caller, and let a second caller silently hijack an existing
worker id by re-registering it with a different token. Adds an optional
pre-shared AdmitToken (ORCHESTRA_FEDERATION_ADMIT_TOKEN) and requires a
same-id re-registration to present the existing worker's own token.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
The 1s ticker's own ExpireLeases call almost always won the race against
the coordinator's 30s Monitor loop, so the coordinator's ExpireLeases saw
nothing left to expire and never reached its session-kill path — silently
orphaning herdr panes past their lease TTL whenever a coordinator is
configured. AUDIT.md called this "harmless" on the assumption CAS merely
picked a winner; the real effect is the pane-kill side effect never firing.
Now the ticker defers reclaim entirely to the coordinator when one exists,
keeping only AssignPending as a periodic retry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Adds an optional "harness" field so codex/opencode completions route to
their own Usage readers instead of always assuming Claude's transcript
format; unblocks the server side named as open in progress.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Amendments to due/description/inherent_priority were accepted and durably
logged but silently discarded by the projection since store.apply only
ever handled the title key. Also added the missing Task.Description field
(TaskCreated never populated it either).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Brief.Git was a single GitSync read from ORCHESTRA_DATA (never a git
checkout), and completions were counted but discarded their report_ref/
receipt. Brief.Git is now keyed by project ID and built from each
project's real repo; GitSync gained Ahead/Behind vs upstream; Brief now
carries Receipts pulled from each TaskCompleted payload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Closes S4 (AUDIT.md): Fanout.Run returned on the first sender error,
permanently ending notifications for the process lifetime after one ntfy
hiccup. Failed sends now go through an OnError hook and the loop
continues. Also persists the delivery cursor to a file next to
ORCHESTRA_DATA so a restart resumes from the last delivered event instead
of re-notifying the entire log from seq 0. Adds the package's first test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Closes Phase 2 items 1-2 (AUDIT.md): Coordinator.TurnDecision evaluates
occupancy/turn-boundary/handoff state synchronously per turn and returns
continue/prepare_handoff/rotate_now/refuse, exposed via POST
/v1/harness/turn. The Claude Stop hook now calls it on ordinary turn
boundaries instead of no-op'ing, and exits 2 on refuse.
Also closes B7's post-hoc producer: /v1/harness/complete now appends a
QuotaReported event from the completing lease's harness usage, so the
router's quota-availability filter and the brief's quota_consumed stop
evaluating against a permanent zero. Live per-harness push producers
(Claude statusline, Codex rollout tail) remain unbuilt — investigation
recorded in AUDIT.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
MarkdownChanges was deleted as dead code, but the underlying spec
requirement wasn't abandoned — rebuilt it independently. Adds
continuity.ConventionsHash for AGENTS.md/CLAUDE.md/VOCAB.md, tracks a
per-session snapshot on herdr.Session, and adds
Coordinator.checkConventions (run every Monitor tick) which compares
each active session's snapshot against its project's base repo and
pushes an in-pane notice via a new herdr.ConventionsNotifier
capability when they drift.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Release already validated and uploaded a §6.1 handoff, but nothing ever
told the agent the .orchestra-handoff.json convention existed, so the
file it waited on never got written. rotate() now prompts the agent
once via a new optional herdr.HandoffRequester capability
(CLIAdapter.RequestHandoff) when the file is missing, and defers
Release until it appears, mirroring the .orchestra-report.md/B3 ask
pattern rather than inventing a handoff.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Release now re-verifies every handoff Anchor.Dirty file hash (previously
unchecked after the top-level anchor SHA compare), snapshots dirty state
onto a per-task scratch branch before uploading, and rewrites the anchor
to the new commit so successor pickup collapses to a single HEAD compare.
ScratchCommit made idempotent for repeated rotations of the same task.
Bootstrap's prompt now points the agent at the scratch-branch commit
history instead of vague "read the handoff" prose, and does not claim a
GET /v1/artifacts/<ref> endpoint that doesn't exist.
MarkdownChanges had zero callers and zero tests; deleted per AUDIT.md's
explicit deletion option rather than half-wiring an undesigned feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Fixes AUDIT.md's B6: nothing wrote a TASK.md into a worktree, so
continuity.ValidatePickup had no caller and no file to check.
- continuity.RenderTaskFile/TaskFileHash: render and hash the immutable
§6.2 TASK.md from a domain.Task.
- GitWorktrees.Create writes and commits TASK.md into every freshly
created worktree (must be committed, not dirty, for ScratchCommit's
immutability check and for a stable hash).
- Coordinator.Start now runs continuity.ValidatePickup (anchor SHA,
dirty-file hashes, TASK.md hash) against the real worktree before
bootstrapping a successor onto a handoff_ref, and blocks the task
instead of bootstrapping on a validation failure.
Still open from Phase 4: handoff production (agent writing the real
handoff; Release still refuses per B5), ScratchCommit wiring before
release, and the §6.2 bootstrap-prompt rewrite — see AUDIT.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
CLIAdapter.Lease's initial prompt used wait=0, skipping the inline wait
Bootstrap already used. The spec (§5.1) requires inline wait on
agent.prompt for bootstrap injection so the prompt isn't sent into a
half-rendered pane.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
POST /v1/harness/complete lets a Claude Code Stop hook report task
completion instead of relying on a human hitting the manual endpoint.
The hook only fires on an explicit .orchestra-report.md marker (not
every turn boundary); the server builds the receipt itself from the
real transcript via herdr.ClaudeUsage rather than trusting a
self-reported number. Codex/opencode producers and the turn-decision
endpoint are still unbuilt — see AUDIT.md/progress.md for scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
Verified against a live herdr instance (192.168.1.105:9245) that pane.kill,
pane.release, and pane.rotation_signal never existed in the protocol, as
AUDIT.md's B5 suspected. Real method list captured in deploy/herdr-schema.json.
- Kill now calls the real pane.close({pane_id}).
- RotationSignal interface/method/call-site deleted; no real equivalent exists.
- Release now refuses loudly instead of calling a nonexistent method — the
real pane.release_agent can't return a handoff_ref either way (herdr
doesn't write handoffs, the agent does), so a real fix needs Phase 4
handoff production first.
Also documents Phase 0 findings in AUDIT.md/progress.md, and adds
CLAUDE.md/AGENTS.md with project-specific knowledge (herdr protocol facts,
deployment topology, a currently-stuck live task, the federation fork) for
future sessions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
S5: Store.Lease and Store.ExpireLeases both set Event.ID to the task id, so
every TaskLeased/TaskReleased event for a given task collided on ID across
every lease of that task — unsound for ApplyAdvisory or any future
ID-based lookup. Both now call domain.NewID().
S6: Append's TaskCreated dedup path returned nil (success) without
appending anything. main.go's handler then did
`s.Events(0)[len(s.Events(0))-1]` and returned that — an unrelated event —
with 201 Created, and every other Append caller (Gitea poll/webhook, JSONL
ingest) had no way to distinguish "duplicate, as expected" from "genuinely
appended".
Add domain.ErrDuplicate, returned instead of nil on a duplicate
(source, external_id). Add Store.TaskBySource to resolve the
already-ingested task by that same dedup key. Update every caller:
- main.go's POST /v1/tasks now returns 200 with the existing task on
ErrDuplicate instead of fabricating a 201 with the wrong event.
- provider.Gitea.Poll/IngestWebhook and provider.JSONL.Ingest treat
ErrDuplicate as expected (already-seen issue/line), not a failure —
without this, Gitea polling would have errored out of its loop on the
first already-ingested issue in every batch, since Poll previously
relied on the old nil-on-dup behavior to keep scanning.
TestLeaseAndExpireEventIDsAreUnique and TestTaskBySourceResolvesDuplicate
cover the store-level fixes; TestAppendReplayAndDeduplicate updated for the
new error signal.
AUDIT.md S5, S6.
Router.attempts (checked against RetryPolicy.MaxAttempts) advanced on
every TaskReleased and again on every subsequent lease. Rotation is
TaskReleased carrying a valid handoff_ref (spec §5.3: "rotation =
intra-task lease transfer"), not a failure — so a task healthy enough to
rotate twice hit the default MaxAttempts=3 and was killed by the retry
limit meant for genuine failures (expiry, crash).
HandleEvent now only advances attempts (and applies retry backoff) for a
TaskReleased whose payload lacks a handoff_ref. AssignPending no longer
increments attempts at lease time at all — that was double-counting
against the same failure that a subsequent non-rotation release already
counts.
TestRotationDoesNotCountAgainstRetryLimit drives a task through 5
rotate-and-release cycles with MaxAttempts=3 and asserts it never reaches
TaskFailed.
AUDIT.md B4.
Brief.From/To both serialized as "from" (repeated tag key on one combined
field declaration), and GitSync.Branch/Head/Status all serialized as
"branch" — Go only honors the first `json:"..."` tag on a struct field, so
the second and third tags in `json:"branch" json:"head" json:"status"` were
silently ignored. go vet ./... failed on this; the brief's git state was
unparseable by any client since Head and Status never appeared in the JSON
at all and From/To collided.
Split the combined declarations so each field gets its own tag. go vet
./... now passes clean.
AUDIT.md S1.
System means "the plane itself, in-process" (router, coordinator, adapters,
lease-expiry reclaim) and is unconditionally FullControl with no token
gate. But it was reachable straight from the X-Orchestra-Surface HTTP
header, both in authz.HTTP's token check and in main.go's own `surface`
closure (which every handler actually calls to authorize an event — it
re-parses the header independently of what the HTTP middleware resolved).
Since no deployment configures ORCHESTRA_SYSTEM_TOKEN (no legitimate HTTP
caller should ever need one), tokens[System] is always "", so the token
check was skipped entirely: any LAN request with
"X-Orchestra-Surface: system" got unauthenticated full control to emit any
event on any task.
Both the authz.HTTP middleware and main.go's `surface` closure now
downgrade System to Web before doing anything else with it, so the header
can never resolve to System over HTTP regardless of token config.
AUDIT.md B8.
CLIAdapter.Occupancy called a.Usage(s.PaneID), but ClaudeUsage/CodexUsage/
OpenCodeUsage all take a filesystem path to session state, not a herdr pane
id. Every call failed with "open <pane-id>: no such file", and
Coordinator.rotate silently `continue`d past every failure, so occupancy
always looked unmeasurable and rotation never fired.
Add herdr.Session.SessionFile and CLIAdapter.resolveSessionFile:
- claude: ClaudeSessionFile resolves the transcript by newest-mtime under
Claude Code's own encoded project directory
(~/.claude/projects/<abs-worktree-with-/-as-minus>/*.jsonl). This is the
Phase-1 fallback; the Stop hook's transcript_path (Phase 2) is the
authoritative source once wired.
- codex: routes through the existing CodexActiveUsage sqlite/rollout
discovery instead of the pane id.
- opencode: resolution needs a live session id from the SSE/status API,
not derivable from the worktree alone — refuses loudly with a pointer
to AUDIT.md Phase 1 rather than guessing a path, per the spec's "verify
against a live session before wiring any trigger" (§5.2.1).
A missing/unreadable session file is now a hard error, not a silent
zero-usage Usage{}. SessionHealth gained Occupancy/OccupancyError fields,
populated every refreshSessionHealth tick, so GET /v1/tasks/{id}/health
makes the number rotation decides on observable before trusting it.
Tests: TestClaudeUsageIsLastTurnNotCumulative guards the exact trap named
in §5.2.1 (large early-turn total, small last-turn usage -> low occupancy).
TestClaudeSessionFileNewestByMtime and TestClaudeSessionFileMissingIsHardError
cover the resolver.
AUDIT.md B1. Live verification against a real Claude Code session (the
spec's own acceptance bar for this phase) still needs to happen on a host
with an actual session — not possible from this sandbox.
AdapterFactory.Herdrs is keyed by herdr instance id (e.g. "homesrv-claude"),
but Reconcile, expire, and rotate all looked adapters up by session.Harness
(the harness kind, e.g. "claude"). In production this key never resolves,
so every one of those call sites silently no-ops via a bare `continue`:
orphaned panes are never killed on restart, expired leases never kill their
pane, and rotation exits before it begins.
Add Coordinator.adapterFor(taskID, session), matching the fallback already
used correctly by refreshSessionHealth (HerdrID, then the lease's
HarnessID, then Harness as a last resort), and route all four call sites
through it.
Regression test TestAdapterResolvedByHerdrIDNotHarnessKind registers an
adapter under "homesrv-claude" and leases with Session.Harness == "claude"
(reproducing the real key mismatch) and asserts rotation still fires — the
existing rotation tests used a keyed-by-nothing fake adapter that matched
any lookup string and so masked this bug entirely.
AUDIT.md B2.
Pre-existing uncommitted work found at session start: rotation now emits
anchor_sha on TaskReleased (previously silently dropped by store.Append
validation), multi-repo Gitea provider support, per-project git worktree
roots, and associated test coverage. Committing as a checkpoint before
starting remediation work tracked in AUDIT.md.
kami