Record what an operator repaired, and let debt count incidents

Slice B, second half. OperatorInterventionRecorded is the one command for
saying "I fixed this by hand": a manual repair happens outside Orchestra by
definition, so the only honest way to have the evidence is for the person
who made it to state it. Inferring "an operator probably intervened" from a
gap would put guesses into the record the ledger is built from.

The debt projection now consumes both new kinds. A closed incident is one
observation carrying its repeat count as intensity, so recurrence stays a
count of independent incidents: 301 repeats on one lease and 2 on another
is a recurrence of two with an intensity of 303, not a recurrence of 303.

Both kinds were previously reported as holes in the system. They are
ordinary evidence now, so their absence from a history is a fact about that
history, and the gap list says so.

The worker also stamps a per-process incarnation on registration and every
heartbeat. Nothing else on the wire distinguishes a restarted worker from a
running one, and an incident cannot outlive the process that reported it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
This commit is contained in:
2026-08-30 06:53:26 +04:00
parent 438c1d6df3
commit e0601296e0
9 changed files with 337 additions and 29 deletions
+11 -4
View File
@@ -21,10 +21,14 @@ type Worker struct {
Capacity int `json:"capacity"`
SupportedProjects []string `json:"supported_projects"`
Build buildinfo.Info `json:"build"`
LastSeen time.Time `json:"last_seen"`
Online bool `json:"online"`
Health WorkerHealth `json:"health"`
Token string `json:"-"`
// Incarnation identifies one worker process. Nothing else on the wire
// distinguishes a restarted worker from a running one, and an observation
// incident cannot outlive the process that reported it.
Incarnation string `json:"incarnation,omitempty"`
LastSeen time.Time `json:"last_seen"`
Online bool `json:"online"`
Health WorkerHealth `json:"health"`
Token string `json:"-"`
}
// WorkerHealth is reported by the worker that owns the local execution backend.
@@ -42,6 +46,9 @@ type WorkerHealth struct {
// Observations is the bounded set of distinct failures behind LastError,
// which keeps its wire name and still reports only the newest.
Observations []Observation `json:"observations,omitempty"`
// Incarnation repeats the worker's process identity on every heartbeat, so
// the coordinator sees a restart even if it missed the registration.
Incarnation string `json:"incarnation,omitempty"`
}
// Observation is one distinct worker failure with its repeat count. A single