Report a bounded ring of worker failures, not one slot
F18. A single last_error slot destroyed causal evidence twice. Run 7 kept only the last of four failures. In run 11 a five-second retry loop on a dead task pinned the slot for twenty-six minutes, so the live task's own expiry was never visible at all, and run 12 lost diagnosis time to the same thing before F58 removed the flood. WorkerHealth now carries up to sixteen distinct observations, each with its repeat count and first/last times. Collapsing is by message rather than by position, because a loop interleaved with other failures would otherwise still flush the ring. Eviction drops the least recently seen. last_error and error_at keep their wire names and still report only the newest failure, so nothing reading them has to change. The ring lives in memory beside last_error and is not persisted, which is the behaviour last_error already had across a restart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
This commit is contained in:
@@ -39,6 +39,21 @@ type WorkerHealth struct {
|
||||
ActivePane string `json:"active_pane_id,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
ErrorAt time.Time `json:"error_at,omitempty"`
|
||||
// Observations is the bounded set of distinct failures behind LastError,
|
||||
// which keeps its wire name and still reports only the newest.
|
||||
Observations []Observation `json:"observations,omitempty"`
|
||||
}
|
||||
|
||||
// Observation is one distinct worker failure with its repeat count. A single
|
||||
// last_error slot let one five-second retry loop erase the cause of everything
|
||||
// around it: run 7 lost three of four failures, and in run 11 the slot was
|
||||
// pinned to a different, blocked task for twenty-six minutes. Repeats collapse
|
||||
// here so a loop cannot evict the failures beside it.
|
||||
type Observation struct {
|
||||
Message string `json:"message"`
|
||||
Count int `json:"count"`
|
||||
First time.Time `json:"first"`
|
||||
Last time.Time `json:"last"`
|
||||
}
|
||||
|
||||
// Capture is published by a worker that owns the pane. The coordinator never
|
||||
|
||||
Reference in New Issue
Block a user