fix(delivery): stop killing the fanout goroutine on a single send error

Closes S4 (AUDIT.md): Fanout.Run returned on the first sender error,
permanently ending notifications for the process lifetime after one ntfy
hiccup. Failed sends now go through an OnError hook and the loop
continues. Also persists the delivery cursor to a file next to
ORCHESTRA_DATA so a restart resumes from the last delivered event instead
of re-notifying the entire log from seq 0. Adds the package's first test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1rkJ2hBMybnJctPbcy4tT
This commit is contained in:
kami
2026-07-27 23:19:15 +04:00
parent 972845bd98
commit 1f46a34afb
5 changed files with 150 additions and 5 deletions
+38 -3
View File
@@ -172,9 +172,44 @@ Fixed so far:
in-pane prompt to write the file before exiting, which is a live-deployment
fact, not something provable from source.
Not yet started: B7 (quota projection has no producer),
Codex/opencode completion producers, the turn-decision endpoint, S2S4,
S7S11. See `AUDIT.md` for the full plan.
- **B7 (post-hoc producer) + Phase 2 turn-decision endpoint** — landed
together, since both are new `QuotaReported`/turn-boundary paths off the
same completion/turn events. `POST /v1/harness/complete` now appends a
`QuotaReported` event (`harness_id` from the closing lease, `consumed`
from the same `usage.Numerator()` used for the receipt), so the router's
5h/weekly availability filter and the brief's `quota_consumed` stop
evaluating against a permanent zero. New `Coordinator.TurnDecision`
(`internal/orchestrator/orchestrator.go`) mirrors `rotate()`'s per-task
logic (occupancy → turn-boundary → handoff-file → release) but runs
synchronously once per turn instead of waiting for `Monitor`'s ticker,
returning one of `continue`/`prepare_handoff`/`rotate_now`/`refuse` via the
new `POST /v1/harness/turn`. The Claude Stop hook
(`deploy/hooks/orchestra-stop.sh`) now calls this endpoint on every
ordinary turn boundary (report marker absent) instead of no-op'ing, and
exits 2 on `refuse` to stop the harness from finishing an unsafe turn.
Covered by `TestTurnDecision` (`internal/orchestrator/rotation_test.go`):
continue-below-threshold, refuse-when-not-at-boundary, and
rotate_now-releases-and-emits-a-valid-TaskReleased cases.
**Not done:** live per-harness *push* producers (Claude statusline,
Codex rollout tail) that would give B7 a second, continuous producer
independent of task completion — recorded as a design investigation in
AUDIT.md ("Real harness quota sources") but not implemented; Codex/
opencode's own equivalents of the Claude Stop hook (whether their
turn-boundary mechanism actually calls `/v1/harness/turn`) also remain
unbuilt, same caveat as Phase 2 item 4 already named for `/complete`.
- **S4** — `delivery.Fanout.Run` used to `return` on the first sender error,
permanently killing the notification goroutine (a single ntfy hiccup meant
no notifications for the rest of the process's lifetime, since nothing
restarts it). Failed sends now go through an `OnError` hook instead of
aborting the loop. Cursor is also persisted now (`SaveCursor` → a
`delivery-cursor` file next to `ORCHESTRA_DATA`, loaded on startup), so a
restart resumes from the last delivered event instead of re-notifying the
entire log from seq 0. `internal/delivery` previously had zero tests;
added `TestFanoutContinuesAfterSendError`.
Not yet started: Codex/opencode completion producers, S2S3, S7S11. See
`AUDIT.md` for the full plan.
**Phase 0 done (2026-07-27):** this box has live TCP reachability to the real
herdr instance at `192.168.1.105:9245` — verified by hand (raw JSON-RPC