430 lines
36 KiB
Markdown
430 lines
36 KiB
Markdown
# [ORCHESTRA] — Multi-Harness Agent Orchestration
|
||
|
||
> Working name; yours to set. This is the binding spec — implementers (agents or you) bind to it, not to prose in a chat log.
|
||
|
||
**Status legend**
|
||
`[S]` settled — decided, do not re-litigate.
|
||
`[D]` default — a value/choice not explicitly ratified; tunable, veto freely. Never treat as fact.
|
||
`[O]` open — unresolved. Has a degrade-safe default so a gap never becomes a hallucinated decision.
|
||
|
||
---
|
||
|
||
## 0. Orientation
|
||
|
||
**What this is.** A control plane for an *orchestra of opaque harnesses* — Claude Code, Codex, opencode(-zen/go), and local models via llama-server. Subscription access is CLI-only: you orchestrate the **process**, not the model. Every harness is a black box with a terminal; herdr is the grip on it.
|
||
|
||
**What this is NOT.** `[S]` This is not Correx. Correx drives models you *call* (local inference, direct API). This drives harnesses you *cannot call*. They share **ontology** (task schema, vocab, the `*.md` conventions) so Maven speaks one language to both — they do **not** share execution code. The layers below have nothing in common with Correx's proposal-validator plane except discipline.
|
||
|
||
**The one metric.** `[S]` The objective is **maximized unattended runtime** — work completed per stretch you can walk away from — not cost-per-turn. Babysitting is a real cost that never appears in `ccusage`. Every trade-off resolves against this: Opus for correlation work cheaper models can't do, local/free for loud-failure mechanical work, rotation to extend the run before the context wall ends it.
|
||
|
||
**Deploy target.** `[S]` Runs on both machines (server, workpc). Single-user, homelab-internal.
|
||
|
||
---
|
||
|
||
## 1. Invariants — must hold everywhere
|
||
|
||
These are asserted at every boundary, not implemented as a component. If a change violates one, the change is wrong.
|
||
|
||
1. **`[S]` The log is append-only, forever.** Corrections are *compensation events* appended on top. There is no rewrite/compaction path — you cannot replay a decision you deleted, and receipts are the point.
|
||
2. **`[S]` The plane emits events, not the agent.** The agent only writes code and artifacts. The router, the adapter, and the stop-hook/wrapper emit lifecycle events. The agent is not a participant in the task system — the harness is. (This is what keeps task tooling out of the agent's context.)
|
||
3. **`[S]` Events reference artifacts by hash; they never embed them.** Handoffs, reports, diffs → CAS. The event carries the hash. Event payloads are always small.
|
||
4. **`[S]` No free-form field is trusted as instruction.** Every event payload and artifact field is typed and schema-validated at write. Knowledge/handoff prose is framed and consumed as **data**. This is the injection boundary — saturated on purpose: assert it everywhere, never build it as one guarded component.
|
||
5. **`[S]` State lives in artifacts verifiable against ground truth — never in agent memory or prose.** Pickup-without-rediscovery works only because a fresh agent validates against repo + `TASK.md`, never against a predecessor's narrative.
|
||
6. **`[S]` The log is not the transcript.** Liveness and per-turn progress are answered by asking herdr / exit codes, not by replaying events.
|
||
|
||
---
|
||
|
||
## 2. Architecture
|
||
|
||
Four layers. Each reads from the one below. Conflating them is the trap.
|
||
|
||
```
|
||
┌─ Layer 4 · SURFACES ──────────────────────────────────────┐
|
||
│ tui/web (full) · telegram+ntfy (notify) · Maven (gated) │
|
||
│ authorization enforced on the bus, not per-client │
|
||
└───────────────▲───────────────────────────────────────────┘
|
||
│ subscribe / emit (per authz)
|
||
┌─ Layer 3 · CONTINUITY ────────────────────────────────────┐
|
||
│ handoff schema · pickup contract · TASK.md · shared *.md │
|
||
└───────────────▲───────────────────────────────────────────┘
|
||
│ artifacts (by hash)
|
||
┌─ Layer 2 · HARNESS (execution) ───────────────────────────┐
|
||
│ herdr backbone · harness adapters · router · rotation │
|
||
│ supervises opaque CLI processes; no API in the loop │
|
||
└───────────────▲───────────────────────────────────────────┘
|
||
│ consumes tasks / emits lifecycle
|
||
┌─ Layer 1 · SUBSTRATE (the sink) ──────────────────────────┐
|
||
│ canonical event log · provider port · task aggregate │
|
||
│ event catalog = THE contract (§4) │
|
||
└────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
**Build order is bottom-up (§8).** Layer 1 is build-ready now; higher layers carry marked open items.
|
||
|
||
### 2.1 Deployment topology `[D]` (parse of a federated design — verify)
|
||
|
||
`[S]` **Federated, not single-host.** Agents run on **both** homesrv and workpc, on demand — but workpc is **not** always on, so this is a worker-federation, not a symmetric pair.
|
||
|
||
Two registries:
|
||
- **Machine registry** — hosts (`homesrv` always-on; `workpc` intermittent; +future), each with mesh address + reachability.
|
||
- **Herdr registry** — herdr instances, each **bound to a machine-registry entry**, carrying static per-instance config for what it may run.
|
||
|
||
`[S]` **homesrv is authoritative.** Event log + router + CAS live there and **only** there — a laptop that sleeps can't hold canonical truth. `[S]` **workpc is a worker + client**, never a source of task truth.
|
||
|
||
`[S]` **herdr on both; we drive it, we don't own it.** The router dispatches leases to **any registered + reachable** herdr per its static config. A herdr that's offline (workpc asleep) simply isn't a candidate that cycle — no special-casing.
|
||
|
||
`[S]` **Git is the cross-machine transport.** Server holds the remotes; a machine going offline has already pushed, a machine coming online pulls. **This makes the scratch-branch commit (§6.2) one mechanism for three handoffs — cross-rotation, cross-agent, and cross-machine.** State in git + a validated artifact, picked up by whoever/wherever.
|
||
|
||
| Component | Host | Notes |
|
||
|---|---|---|
|
||
| event log · router · CAS | **homesrv only** | canonical; always-on |
|
||
| herdr + leased harnesses | homesrv **and** workpc | worker federation; offline = not a candidate |
|
||
| git remotes | homesrv | cross-machine transport |
|
||
| tui / web control | workpc (and anywhere) | client |
|
||
| Maven | homesrv | fully server-side (§2.2) |
|
||
|
||
### 2.2 Projects & machine affinity `[S]`(model) `[D]`(examples)
|
||
|
||
`[S]` **Tasks bind to a project at creation** — you pick which. `[S]` Projects are **first-class and extensible**: adding more later reshapes nothing (the binding is a field + a config entry, not a schema change). `[S]` A project declares **machine affinity** — which host(s) its harnesses run on — as **static manual config** ("work however it sees fit based on static configuration").
|
||
|
||
Resolution chain: **task → project → affinity → candidate machines → router filters by reachable+configured herdr there.**
|
||
|
||
`[D]` Illustrative affinities (yours, not the system's — shown to fix the model):
|
||
|
||
| Project | Affinity |
|
||
|---|---|
|
||
| correx | workpc-primary, git remote on homesrv |
|
||
| Maven | homesrv-full; workpc only for CPT / fine-tune / LoRA |
|
||
| Kdrive · Muzic · Nexus · Praxis · Hexis | homesrv-exclusive |
|
||
|
||
`[O]` **Fork — is affinity hard or soft?** Night queue on homesrv: a workpc-primary project (correx) has its remote on the server, so homesrv *could* run it against the remote — or you may want it to wait for workpc. **Default `[D]`: hard** (never run where not configured; overnight queue = server-runnable projects only) — safer, predictable, matches static config. Soft (prefer workpc, fall back to server overnight) is a one-flag change if you want it. See §9.
|
||
|
||
### 2.3 Workflow — end to end `[S]`
|
||
|
||
Two walk-away modes, both reducing to the same lifecycle:
|
||
|
||
- **Mode A — mid-day handoff.** workpc is on; you queue/assign and walk away; workpc's herdr runs to completion; results commit + push to the homesrv remote; `TaskCompleted` per task. (You were just using workpc, so it's up — no sync-wait.)
|
||
- **Mode B — overnight batch.** You queue N tasks for the night on homesrv (server-runnable projects, per affinity above); homesrv's herdr works through them; each commits + pushes; **git-sync means workpc pulls in the morning**; a single **morning brief (§7.4)** rolls up the whole night. This is the payoff of the sync + brief machinery.
|
||
|
||
**The whole lifecycle, once:**
|
||
|
||
```
|
||
ingest → TaskCreated (provider adapter; §3.2, §4)
|
||
→ router: project → affinity → machine → lease (§2.2, §5.5) ── offline herdr = skipped
|
||
→ herdr on that machine: bootstrap · work · gate · TDD (§5, §6)
|
||
→ rotation as needed (intra-task lease transfer; state via git) (§5.3, §6.2)
|
||
→ TaskCompleted + report_ref (proofs + receipts) (§4)
|
||
│ or → TaskFailed / TaskBlocked → you (needs-human)
|
||
→ reflect status out · git push · (workpc pulls on wake) (§2.1, §3.2)
|
||
→ brief rolls up the window → surfaces per authz (§7.4, §7.1)
|
||
```
|
||
|
||
`[S]` Everything crossing a machine boundary is git + a validated artifact — never live state over the wire. The event log is the only always-on shared surface; nothing holds a second copy.
|
||
|
||
---
|
||
|
||
## 3. Layer 1 — Task Substrate `[S]` (spine settled)
|
||
|
||
### 3.1 Canonical event log
|
||
|
||
`[S]` The event log is the single source of task truth. Append-only. Global monotonic sequence number = the subscriber cursor. Each task aggregate additionally carries a per-task version for optimistic concurrency (§4).
|
||
|
||
`[S]` **Corrections = compensation.** A wrong event is never edited; a compensating event is appended and replay sees both.
|
||
|
||
`[S]` **Snapshots** cache "state as of sequence N" so projections don't replay from zero. Raw events are retained regardless — snapshots are an optimization, never a substitute.
|
||
|
||
`[D]` **Storage backend.** Append-only log store TBD; CAS blobs can sit on existing MinIO. Choose in §8 spike.
|
||
|
||
### 3.2 Provider port — bidirectional adapter
|
||
|
||
`[S]` The provider is an **interface, not a default implementation**, and it is **not** the source of task truth. It has two directions:
|
||
|
||
- **Ingest:** external task → `TaskCreated` event.
|
||
- **Reflect:** task state transitions → external status (e.g. close the Gitea issue on completion).
|
||
|
||
`[S]` **Stable external key `(source, external_id)`** per adapter, serving double duty:
|
||
- **Idempotency on ingest** — a webhook firing twice, or a poll re-seeing an issue, must not create two `TaskCreated`. First sight creates; repeats no-op or update.
|
||
- **Reflection address** — tells the adapter which external record to update outbound.
|
||
|
||
Adapters (all implement the same port):
|
||
- Gitea issues / CI failures (webhook + poll)
|
||
- Pure-local JSONL (trivial baseline — build first)
|
||
- (later) anything Maven feeds in
|
||
|
||
### 3.3 Task aggregate
|
||
|
||
`[S]` **Task is the only aggregate.** Epic / roadmap / sprint are **projections** over parent links, not their own streams (sprint's open/close lifecycle is the one possible exception — deferred to the agile layer).
|
||
|
||
Fields:
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| `id` | ulid | internal |
|
||
| `source`, `external_id` | str | the stable external key |
|
||
| `project` | project-id | **bound at creation** (§2.2); resolves to machine affinity. Sink and brief group on this. |
|
||
| `capability` | tag set | **static**, set at creation. What the task *needs*. Router matches on this. |
|
||
| `lease` | Lease? | **dynamic**. Who holds it *now*. See §3.4. |
|
||
| `parent` | task-id? | task→epic→roadmap link |
|
||
| `inherent_priority` | enum | raw priority, editable |
|
||
| `due` | instant? | no hard deadlines; feeds derived importance |
|
||
| `estimate` | `{value, who, confidence}` | provenance-carrying (§3.6) |
|
||
| `state` | enum | derived from lifecycle events |
|
||
| `version` | int | optimistic-concurrency guard |
|
||
|
||
### 3.4 Two axes: capability vs. lease `[S]`
|
||
|
||
These are **independent** and must not be merged:
|
||
|
||
- **Capability** — a property of the *task*. Static tag set (`{correlation}`, `{mechanical}`, …). Set at creation, matched at pickup.
|
||
- **Lease** — a property of a *harness instance*. Dynamic: who holds it, and until when (TTL, §5.4). Router sets it, agent releases it, rotation transfers it.
|
||
|
||
Merging them is wrong because it would force recomputing task metadata every time quota shifts — the task didn't change, the harness did.
|
||
|
||
`[S]` **Availability ≠ capability.** Whether a given harness *can accept right now* (alive? quota headroom? under concurrency cap?) is a **runtime filter the router applies at lease time**, not a task field. Router **matches** on capability, **filters** by availability.
|
||
|
||
`[S]` **Pickup-without-rediscovery** = lease release keeps capability intact; the next matching+available harness picks up from task state. Rotation-handoff and agent-to-agent-handoff are **the same lease transfer over the same task state** — one mechanism.
|
||
|
||
### 3.5 Derived importance `[S]`
|
||
|
||
Importance is **not stored**. It is a sort key over the sink: `f(inherent_priority, proximity_to_due)`, recomputed as due approaches. `[S]` **Pickup-only** — importance orders the sink when a lease frees; it **never preempts running work**. No preemption event, no interrupt path in the harness.
|
||
|
||
### 3.6 Agile structure `[S]`
|
||
|
||
- Hierarchy = parent links (§3.3), rendered as projections.
|
||
- Sprint = a time-boxed *selection* over the sink.
|
||
- Estimates carry `{who, confidence}` — agents give weighted judgment; you keep the last word. (This is the "standup" mechanic reduced to a field; see §7 for the advisory-event form.)
|
||
|
||
---
|
||
|
||
## 4. Event Catalog — THE contract
|
||
|
||
`[S]` An event earns a **distinct type only when a subscriber reacts in a way a generic "something changed" could not dispatch.** Everything else is one generic amend.
|
||
|
||
### Behavioral events (distinct subscriber reactions)
|
||
|
||
| Event | Emitted by | Key payload | Reacted to by | Concurrency |
|
||
|---|---|---|---|---|
|
||
| `TaskCreated` | provider adapter (ingest) | `project`, `capability`, `source`, `external_id`, `parent?`, `inherent_priority`, `due?` | router → evaluate for assignment | dedup on `(source,external_id)` |
|
||
| `TaskLeased` | router | `harness_id`, `expected_version`, `ttl` | the matched harness → start work | **CAS on task `version`** — second writer fails, no lock |
|
||
| `TaskReleased` | agent/harness (via plane) | `handoff_ref` (hash), `anchor_sha` | router → find next picker | version-guarded |
|
||
| `TaskCompleted` | stop-hook/wrapper (via plane) | `report_ref` (hash), `receipt` | fan-out: telegram/Gitea/Maven; reflect status out | terminal |
|
||
| `TaskFailed` | router | `reason`, `attempts` | needs-human path; stop retrying | terminal |
|
||
| `TaskBlocked` | agent/harness (via plane) | `blocker`, `handoff_ref?` | needs-human path | — |
|
||
| `ApprovalRequested` | harness (gate/arbiter) | `subject_ref`, `options` | full+gated surfaces present it; harness blocks | — |
|
||
| `ApprovalGranted` / `ApprovalDenied` | full-control surface | `subject_ref`, `by` | harness unblocks; other surfaces learn via this event | resolve-by-event, no timeout |
|
||
|
||
### Generic event
|
||
|
||
| Event | Emitted by | Payload | Reacted to by |
|
||
|---|---|---|---|
|
||
| `TaskAmended` | human or agent | `{fields}`: title, due, description, `inherent_priority` | sink re-projects |
|
||
|
||
`[S]` `TaskAmended` **never** carries lease or lifecycle. That boundary is the whole thing keeping it from rotting into a CRUD blob.
|
||
|
||
### Explicitly NOT in the log
|
||
|
||
- **Per-turn progress, liveness** → herdr socket (`pane.agent_status_changed`, `pane.exited`) + exit codes.
|
||
- **Token spend** → a **separate projection** keyed by `harness+window`, not by task. Per-task cost lands as a `receipt` field on `TaskCompleted`, **summed across the task's lease intervals** (a rotated task spans several sessions → a sum, not a single delta).
|
||
|
||
### The "no not-mine excuses" mechanism `[S]`
|
||
|
||
Don't trust the agent to self-report a lint/format/test issue. The **quality gate** (detekt/ktlint, mypy/ruff, shellcheck, schema-validate) catches it mechanically and emits `TaskCreated` for anything it won't block on — deterministic and automatic. The only residue is a logic bug the agent *notices* but no gate catches → one narrow "file-a-task" affordance, kept small.
|
||
|
||
---
|
||
|
||
## 5. Layer 2 — Harness (execution)
|
||
|
||
### 5.1 herdr backbone `[S]`
|
||
|
||
herdr (rust agent multiplexer; v0.7.x, AGPL-3.0/commercial — matters only if linked, not if shelled out to) gives real turn state where tmux gave none:
|
||
|
||
- `agent.prompt` with inline `wait{until,timeout_ms}` — bootstrap injection + turn-completion wait in **one request**, no gap.
|
||
- `agent.wait` **pins the pane occupant** — a replacement can't satisfy the old wait.
|
||
- `events.subscribe` on `pane.agent_status_changed` — no polling.
|
||
- `pane.report_metadata` (`tokens` map, TTL) — live occupancy gauge in the sidebar once you have the number.
|
||
- `pane.exited` — crash fast-path. `notification.show` — HALT pages you through herdr itself.
|
||
- `worktree.create` — one worktree per rotation chain → anchor validation collapses to one `rev-parse`.
|
||
|
||
`[S]` Install native integrations (`herdr integration install {claude,codex,opencode}`) so "done" is reported, not inferred. `[S]` Socket at `~/.config/herdr/herdr.sock` is unauthenticated → bind harness fleet to the WireGuard mesh, never `0.0.0.0`; on VPS, reconsider entirely. `[S]` Generate the client off `herdr api schema --json`; check protocol version via `ping` before depending on new behavior.
|
||
|
||
`[D]` **Driving herdr — imperative is fine, except two ops.** Poke panes over the socket (send-text / capture-pane-equivalents) for ordinary control and for feeding the TUI/brief — it's a multiplexer you don't own, this is expected. But keep the **two rotation-critical ops** on the semantic primitives: **turn-end** via Face B / `agent_status` (§5.2.1), and **bootstrap injection** via `agent.prompt` with inline `wait` (`agent.wait` pins the occupant so a replacement can't satisfy it). capture-pane polling + send-keys for *those two* reintroduces the exact tmux race — a bootstrap or `/clear` fired into a half-rendered prompt, silently, unattended. Veto only if you've solved that race another way.
|
||
|
||
### 5.2 Harness adapter — the second load-bearing contract `[S]`
|
||
|
||
A harness adapter has **two faces**, and it is the *same seam* as the Layer-1 provider concept applied to execution:
|
||
|
||
```
|
||
interface HarnessAdapter {
|
||
// Face A — lifecycle / IO
|
||
fun lease(task, worktree): Session // start the opaque CLI in a herdr pane
|
||
fun bootstrap(session, handoffRef) // ~200-token prompt: "read handoff, run validate, proceed"
|
||
fun release(session): TaskReleased // checkpoint + emit
|
||
fun kill(session)
|
||
|
||
// Face B — turn-boundary + occupancy detection (per-harness impl; §5.2.1)
|
||
fun onTurnBoundary(cb) // cc: Stop hook (exit 2 refuses turn); codex: rollout tail; opencode: SSE session.status
|
||
fun occupancy(session): Fraction // all three read native session state — NO proxy required
|
||
}
|
||
```
|
||
|
||
`[S]` **cc's Face B** = the Stop hook. Exit **2** blocks stop and feeds stderr back → over-threshold-with-no-valid-handoff refuses the turn until the handoff is written and validates, then exit 0. `[S]` **codex/opencode's Face B** read their own native session state (§5.2.1) — no proxy. The trigger is harness-agnostic **above** the adapter; per-harness **below** it. `[S]` The router and rotation logic never know which harness they're driving.
|
||
|
||
### 5.2.1 Occupancy sources — verified `[S]`/`[D]`
|
||
|
||
`[S]` herdr has no token accounting, but the earlier "proxy for codex/opencode" assumption was wrong: **all three harnesses write usage into local session state; none require an interception proxy.** They differ only in the cleanest access path. Build this measurement and verify it against a live session **before** wiring any trigger — the whole rotation system rests on this number.
|
||
|
||
| Harness | Source `[S]` | Path / endpoint | Per-turn shape `[S]` | Occupancy numerator (what's in the window *now*) |
|
||
|---|---|---|---|---|
|
||
| **Claude Code** | session JSONL, `message.usage` per assistant entry | `~/.claude/projects/<enc-path>/<session-id>.jsonl` (hook gives `transcript_path` on stdin) | **absolute per entry**: `input_tokens`, `cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens` | last assistant entry: `input + cache_read + cache_creation` (the context sent). Dedupe by message `uuid`. |
|
||
| **Codex** | rollout JSONL `event_msg`, `payload.type=="token_count"` | `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` (or `CODEX_HOME`). Active file authoritative via `state_*.sqlite → threads.rollout_path` — discover, don't hardcode | `info.total_token_usage` is **cumulative** (diff successive); `info.last_token_usage` is the last step | `last_token_usage.last_input (+ last_cached)`. Only present since codex ≥ 2025-09-06; older logs have none. |
|
||
| **opencode** | server API (it's already a Hono server on `:4096`) | SSE `GET /event` → `session.status` (idle/busy) + `message.updated`; or `GET /session/{id}/messages`; files at `~/.local/share/opencode/storage/message/` (`OPENCODE_DATA_DIR`) | per-message `tokens = {input, output, reasoning, cache:{read,write}}` (`cost:0` in files) | last message: `tokens.input + tokens.cache.read`. |
|
||
|
||
`[S]` **The trap that would silently break rotation: never use the cumulative session total as occupancy.** All three accumulate `cache_read` per turn without bound — one real opencode session reported ~56M "total tokens" against a 200k window; codex burns from 21k→560k cumulative in a startup loop. Cumulative total answers "what did this session cost," not "how full is the context." **Occupancy = *this turn's* input+cache against the model's context window**, i.e. the numerator column above ÷ window (200k Sonnet/Opus-class; the gpt-5.x window for codex — model-dependent constants, keep them in config).
|
||
|
||
`[D]` **Proxy is the fallback, not the primary.** A localhost shim on `ANTHROPIC_BASE_URL`/`OPENAI_BASE_URL` reading `usage` off responses still works and is the one *uniform* implementation across all three — worth it only if per-harness parsing gets annoying or you want request-time-synchronous truth. Native sources are simpler and are the default.
|
||
|
||
`[D]` **opencode SSE is not rock-solid** — the `/event` stream has known bugs (closes right after `server.connected`; stochastic REST behavior under `opencode serve`). Treat the event stream as the fast path and keep the message-file / `opencode stats --json` read as the backstop, same fast-path-plus-backstop shape as `pane.exited`+TTL.
|
||
|
||
### 5.3 Rotation `[S]`
|
||
|
||
`[S]` **The Stop-hook/Face-B decides rotation, not the router.** It has the transcript and fires at turn boundaries. The router decides whether rotation is *permitted* (caps, budget) and executes it.
|
||
|
||
Triggers:
|
||
- `[D]` **soft ~55%** → "prepare handoff at next natural stop"
|
||
- `[D]` **hard ~75%** → rotate now; reserve `[D]` ~10–15k tokens to actually write the handoff (do not wait for 90% — compaction beats you)
|
||
- `[S]` **milestone** → phase boundary rotates regardless of occupancy (a fresh agent at a boundary beats a 60%-full one; cheap context is a feature)
|
||
- `[S]` **thrash** → N failed test runs / same file edited M times with no pass / identical tool calls repeating → rotate with `reason=thrash` + populated `dead_ends` = circuit breaker
|
||
- `[S]` **agent-initiated `ROTATE`** → emitted when a coherent unit finishes and the next is independent
|
||
|
||
`[S]` **Rotation = intra-task lease transfer** (§3.4). `[S]` **Split-then-close** (cheap): new pane validates the anchor before the old pane closes — atomic, with a real rollback path, and it drops the `/clear`-slash-command dependency that codex/opencode don't share.
|
||
|
||
### 5.4 Failure & lease semantics `[S]`
|
||
|
||
- `[S]` **Lease carries a TTL.** Reclamation = **lease-expiry**, not crash-detection. A harness that dies without releasing lets its lease go stale → task returns to the sink. You don't need to reliably observe death; you need leases to not be eternal.
|
||
- `[S]` `pane.exited` = fast-path to relet sooner; TTL = backstop for the hang `pane.exited` misses.
|
||
- `[S]` Policy: **retry N, then `TaskFailed`**; agent continues if nothing more important is queued. `[D]` `N`, `ttl`, retry backoff = config.
|
||
|
||
### 5.5 Router `[S]`
|
||
|
||
Makes assignment decisions at **exactly two moments**: on `TaskCreated` (ingest) and on lease-free (`TaskReleased`/expiry). **Never against running work** (pickup-only, §3.5). Algorithm: resolve **project → machine affinity** (§2.2) to candidate hosts → **restrict** to registered + **reachable** herdr on those hosts (offline = not a candidate) → **match** by `capability` → **filter** by availability (alive, quota headroom, under concurrency cap) → **order** by derived importance → lease the top via version-guarded `TaskLeased`. `[S]` Machine resolution is first because it's the hard constraint; capability and availability narrow within it.
|
||
|
||
### 5.6 Build note `[S]`
|
||
|
||
Orchestrator in **bash first** (~150 lines: subscribe, read request, validate, split, prompt, ack, close) — run it on one real task, find where the state machine is wrong (it will be, around thrash + timeout). **Then** port to Kotlin/Spring as a proper long-lived supervised service and slot it into the fleet. Hooks stay bash forever (they run in the agent's process every turn — no runtime, no deps).
|
||
|
||
---
|
||
|
||
## 6. Layer 3 — Continuity
|
||
|
||
### 6.1 Handoff schema `[S]`
|
||
|
||
Every field **verifiable against the repo** — prose is where hallucination lives and it compounds across rotations. `[D]` **Size cap ~4–6k tokens**; if it doesn't fit, task decomposition is wrong, not the schema. Stored in CAS; events carry the hash (Invariant 3).
|
||
|
||
```toml
|
||
[meta] id, parent, reason = "threshold|milestone|thrash|manual", rotation_index
|
||
[anchor] git_sha, branch, dirty = [{path, sha256}] # verifiable state
|
||
[objective] goal (one sentence), done_when = [checkable, criteria]
|
||
[progress] completed = [{what, evidence = "file:line"}], remaining = [...]
|
||
[next] action (single step), command (literal first cmd), files = [...]
|
||
[knowledge] invariants = [...], dead_ends = [{tried, why_failed}], open_questions = [...]
|
||
[verify] build, test, last_result = {command, exit_code, at_sha}
|
||
```
|
||
|
||
`[S]` **`dead_ends` is the highest-value block and the one everyone omits** — negative knowledge is exactly what's lost on rotation; without it the fresh agent re-derives the same failure. `[S]` **`[knowledge]` is framed as data, not instructions** (Invariant 4) — this is the injection-persistence surface; strict schema, no free-form.
|
||
|
||
### 6.2 Pickup contract — the make-or-break `[S]`
|
||
|
||
`[S]` Agents validate; the orchestra validates. `[S]` The validation is a **concrete, testable procedure** run before the fresh agent continues — not a principle:
|
||
|
||
```
|
||
validate-handoff <id>:
|
||
1. schema-check the handoff (types, required fields, no free-form in [knowledge])
|
||
2. git rev-parse HEAD == anchor.git_sha
|
||
3. for each anchor.dirty: sha256(path) == recorded # or: WIP committed to scratch branch, one sha compare
|
||
4. re-read immutable TASK.md (original spec)
|
||
→ all pass → proceed
|
||
→ any fail → do NOT rotate/continue; agent fixes, re-validate
|
||
```
|
||
|
||
`[S]` **`TASK.md` is immutable**, holds the original spec, is **re-injected every rotation**, and is **never rewritten**. The fresh agent validates against repo + `TASK.md` — **never against the predecessor's handoff prose.** This is the defense against the telephone game across long autonomous runs. `[S]` **Commit WIP to a scratch branch first** so step 3 is one sha compare, atomic and free.
|
||
|
||
### 6.3 Shared memory & vocab `[S]`
|
||
|
||
- `[S]` Shared `*.md` (AGENTS.md, CLAUDE.md, vocab) — **all agents contribute.**
|
||
- `[S]` On update, the **orchestra injects a notice** to agents whose current task is **adjacent** (staleness handled at the orchestra layer, not by trusting cached agent views — same shape as herdr's metadata TTL, one layer up).
|
||
- `[S]` Docs are self-containing, human-readable **and** digestible-at-a-glance, with references where available and mockups at specing stage. The policy SHOULDs ("no not-mine excuses", TDD-by-default, spec-first, cheap-harness-before-manual, QA-mandatory) live **here as convention text + gates**, not as code.
|
||
|
||
---
|
||
|
||
## 7. Layer 4 — Surfaces
|
||
|
||
### 7.1 Authorization model — enforced on the bus `[S]`
|
||
|
||
Not a UX detail — a capability per surface on the event bus. "Who can do what from where" is enforced **once, at the bus**, not per-client.
|
||
|
||
| Surface | Capability | Can emit | Notes |
|
||
|---|---|---|---|
|
||
| **telegram + ntfy** | notify-only | *nothing* | read-side subscriber; pure notification |
|
||
| **tui / web-ui** | full control | any event, incl. approvals | the control seats |
|
||
| **MCP** `[D]` | read + **gated** write | queries freely; task create/amend via `ApprovalRequested` | programmatic surface for external agents/tools; `[D]` promote to full if it's your own trusted client. (MCP-as-*ingest* — Maven feeding tasks — is the §3.2 provider port, separate from this.) |
|
||
| **Maven** | observation + **gated** control | **only** via `ApprovalRequested` → your grant; never direct | north-star assistant, fenced |
|
||
|
||
`[S]` **Clients poll; they do not store.** `[S]` An approval granted on **any** full/gated surface resolves by `ApprovalGranted`/`Denied`; others **learn via the event** later. **No approval timeout** (you may be asleep). `[S]` Artifacts crossing into a **control-capable** surface are **re-validated against schema at that boundary** (Invariant 4), never passed through.
|
||
|
||
### 7.2 Quota transparency `[O]`
|
||
|
||
`[S]` **Quota ≠ occupancy — different numbers, do not conflate.** Occupancy (§5.2.1) is *context-window fill of one session* and drives rotation. Quota here is *subscription-pool consumption across sessions* (the 5-hour rolling + weekly caps) and drives the router's availability filter + 3am safety. Same source files, different accounting: `ccusage blocks --json` gives the 5h-window view; codex even surfaces its own 5h/weekly % in the rollout logs.
|
||
|
||
`[S]` Transparent quota is a **projection** (§4, keyed `harness+window`), fed by the same per-harness session state as §5.2.1 (`ccusage`-style) — no separate proxy needed. `[O]` **Accuracy is unresolved** — each pool is reconstructed, none authoritative, and a 3am autonomous run trusting a wrong "you have headroom" hits a wall mid-task. **Degrade-safe default `[D]`:** treat `[D]` 80% reported as full; the router's availability filter (§5.5) uses the conservative number. Name the failure mode in-system; refine after measuring a real run.
|
||
|
||
### 7.3 Standups `[S]`
|
||
|
||
Scheduled read over the log that emits an **advisory** event (agents provide opinion + weighted judgment). `[S]` **Last word is yours** — the advisory routes through the same approval gate; nothing self-applies.
|
||
|
||
### 7.4 The brief `[S]`
|
||
|
||
`[S]` A **projection over the event log for a window** (nightly, or on demand) — **not** a new event source, not something an agent authors. It aggregates, for that window: `TaskCompleted` / `TaskFailed` / `TaskBlocked` with their `report_ref` proofs+receipts, what **needs you** (pending `ApprovalRequested`, failures, blocks), quota consumed per harness (§7.2), and the **git sync state** (what pushed, what's on which branch, what workpc still needs to pull). Read-only, **digestible at a glance** (the SHOULD).
|
||
|
||
`[S]` **The morning brief is just the overnight window's brief** — the payoff of Mode B (§2.3). Delivered per surface authz (§7.1): telegram gets the headline + ntfy ping, tui/web the full rollup. `[S]` It **surfaces** judgment and attention items; nothing in it self-applies — your call, same gate as everywhere.
|
||
|
||
The per-task report (`report_ref`, §4) is one task's receipts; the brief is the window's rollup **over** those reports. Two levels, one is not the other.
|
||
|
||
---
|
||
|
||
## 8. Build Sequence `[S]`
|
||
|
||
Bottom-up. Each step is a consumer of the one below; do not start a step before its dependency is proven.
|
||
|
||
1. **Task schema + provider port** — one interface, one **JSONL adapter**. (§3)
|
||
2. **Event log + state projection** — prove the read model; pick the log store + CAS backend here. (§3.1, §4)
|
||
3. **`*.md` conventions** — near-zero code; encodes most policy SHOULDs. (§6.3)
|
||
4. **herdr harness + rotation** — as a **consumer** of 1–2. Build the **occupancy measurement first** and verify against a live session (§5.2). Orchestrator in **bash**, then port. (§5)
|
||
5. **Surfaces, one at a time** — each a subscriber with its bus capability (§7). Telegram/ntfy first (read-only, lowest risk) → tui/web → Maven-gated.
|
||
|
||
**Cross-cutting from day one:** Invariants (§1) are asserted as each layer lands — especially append-only (1), plane-emits (2), hash-refs (3), typed-no-freeform (4). TDD by default; QA suites mandatory; cheap harness before any manual debug/test.
|
||
|
||
---
|
||
|
||
## 9. Open Questions — with degrade-safe defaults
|
||
|
||
Named so the spec is written against a known frontier. Each has a default so a gap never becomes a hallucinated decision.
|
||
|
||
| # | Question | Status | Degrade-safe default |
|
||
|---|---|---|---|
|
||
| 1 | **Quota projection accuracy** across cc/codex/opencode (reconstructed, non-authoritative) | `[O]` | Treat `[D]` 80% reported as full; router uses conservative number. Refine post-measurement. |
|
||
| 2 | **Schema-evolution mechanic** — upcast-on-read vs. tolerant-reader | `[O]` (versioned = decided) | Tolerant-reader + versioned envelope until a real migration forces upcasting. |
|
||
| 3 | **Log retention / cold-storage aging** as the log grows | `[O]` (deferred) | Snapshot + keep all raw events online; revisit only when size bites. |
|
||
| 4 | **Storage backend** for append-only log (+ CAS on MinIO?) | `[O]` | Decide in Step 2 spike; JSONL baseline works meanwhile. |
|
||
| 5 | **Cross-surface approval propagation** detail (poll cadence, dedup) | `[O]` (deferrable) | Resolve-by-event, no timeout; clients poll `[D]` on a short interval. |
|
||
| 6 | **Capability tag vocabulary** — the actual tag set beyond `{correlation, mechanical}` | `[O]` | Start with those two; grow from real task triage, not up front. |
|
||
| 7 | **Machine affinity: hard or soft?** (§2.2) — can a workpc-primary project run on homesrv against the remote overnight, or wait for workpc? | `[O]` | **Hard** — never run where not configured; overnight queue = server-runnable projects only. Soft is a one-flag change. |
|
||
| 8 | **Cross-machine lease correctness** (§2.1) — worktree/anchor validation and quota accounting when a lease's git checkout lives on a different host than the router | `[O]` | Git is the only cross-machine transport; validate against the local checkout wherever the harness runs; quota accounted per-host. Prove on the first federated run. |
|
||
|
||
**Decided elsewhere (do not reopen):** compensation over compaction; versioned schema; snapshot+replay; per-harness trigger = adapter Face B; retry-then-relet with TTL reclamation; capability-static vs. availability-dynamic; pickup-only importance (no preemption); surface authorization split; injection as an everywhere-invariant; **federated worker topology (homesrv authoritative, workpc intermittent worker); herdr on both, driven-not-owned; git as cross-machine transport; task→project→machine-affinity; the brief as a windowed projection.**
|
||
|
||
**Explicitly out of scope — deferred by decision, not omission:** cheap-model / free-inference **delegation** (offloading mechanical sub-work to local llama-server or free providers). Concluded orthogonal to this system — **rotation is the quota lever, delegation is at most a trim**, and the real objective is unattended runtime, not per-turn price (§0). Revisit later as a per-task routing hint carried in task metadata, once the substrate runs and a profiled session shows where tokens actually go. Named here so it stays a choice.
|
||
|
||
---
|
||
|
||
*This spec is the contract. Where it says `[O]`, the default holds until you decide — an implementing agent must surface the open item, never silently pick.*
|