Stop the ring's history from manufacturing recurrence

Both defects came from the first real run of slice B against live data, and
neither was visible in a unit test written from the design.

The ring is a bounded history, not a set of live conditions. A quiet timeout
closed an incident, its entry stayed in the ring because nothing evicts it,
and every later heartbeat opened the same incident again: three signatures,
four incidents each, from failures that never happened twice. An incident
now opens only when the entry actually advances past what was already
accounted, and the high-water mark survives the close.

The ring also outlives the work it describes, so attributing its entries to
whatever the worker is running now invented an association. The task is read
out of the message, and only a failure that names no task belongs to the
current lease. An incident that names an older task has no live lease to
bound it, so it closes on quiet timeout rather than on the next epoch change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
This commit is contained in:
2026-08-30 14:20:47 +04:00
parent 74cad5d374
commit 79d20534b5
3 changed files with 128 additions and 4 deletions
+14
View File
@@ -118,6 +118,20 @@ func ObservationSignature(message string) string {
return s
}
// ObservationTaskID reads the task a failure was about out of the message
// itself. The ring is a history: it holds entries from tasks that ended long
// ago, so the worker's currently active task is the wrong answer for most of
// them, and attributing an old failure to whatever is running now would be a
// fabricated association.
func ObservationTaskID(message string) string {
if m := observationTaskID.FindString(message); m != "" {
return strings.ToUpper(m)
}
return ""
}
var observationTaskID = regexp.MustCompile(`(?i)\b[0-9A-HJKMNP-TV-Z]{26}\b`)
func ValidateObservationIncidentOpened(p map[string]any) error {
if id, _ := p["observation_id"].(string); strings.TrimSpace(id) == "" {
return fmt.Errorf("%w: observation_id required", ErrInvalid)