e0601296e0
Slice B, second half. OperatorInterventionRecorded is the one command for saying "I fixed this by hand": a manual repair happens outside Orchestra by definition, so the only honest way to have the evidence is for the person who made it to state it. Inferring "an operator probably intervened" from a gap would put guesses into the record the ledger is built from. The debt projection now consumes both new kinds. A closed incident is one observation carrying its repeat count as intensity, so recurrence stays a count of independent incidents: 301 repeats on one lease and 2 on another is a recurrence of two with an intensity of 303, not a recurrence of 303. Both kinds were previously reported as holes in the system. They are ordinary evidence now, so their absence from a history is a fact about that history, and the gap list says so. The worker also stamps a per-process incarnation on registration and every heartbeat. Nothing else on the wire distinguishes a restarted worker from a running one, and an incident cannot outlive the process that reported it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
267 lines
9.5 KiB
Go
267 lines
9.5 KiB
Go
package domain
|
|
|
|
import (
|
|
"fmt"
|
|
"strings"
|
|
"time"
|
|
|
|
"orchestra/internal/review"
|
|
)
|
|
|
|
// Debt is the projected read model for accumulated cost: the defects,
|
|
// workarounds and operational burdens that real tasks keep paying for. It is
|
|
// derived from the event log rather than written by hand, so a claim in the
|
|
// ledger can always be checked against the events that produced it.
|
|
//
|
|
// Four classes, and no more. A fifth invented at runtime makes the priority
|
|
// model meaningless, because the thresholds in operations.CheckDebtEligibility
|
|
// are stated per class.
|
|
type DebtClass string
|
|
|
|
const (
|
|
// DebtCorrectness is behaviour that is wrong or violates an invariant.
|
|
DebtCorrectness DebtClass = "correctness"
|
|
// DebtOperational works, but repeatedly costs time to diagnose, recover,
|
|
// deploy, observe or operate.
|
|
DebtOperational DebtClass = "operational"
|
|
// DebtStructural is duplication or architecture demonstrably raising the
|
|
// cost of future changes.
|
|
DebtStructural DebtClass = "structural"
|
|
// DebtPolish has no demonstrated cost yet, and never promotes itself.
|
|
DebtPolish DebtClass = "polish"
|
|
)
|
|
|
|
func (c DebtClass) Valid() bool {
|
|
switch c {
|
|
case DebtCorrectness, DebtOperational, DebtStructural, DebtPolish:
|
|
return true
|
|
}
|
|
return false
|
|
}
|
|
|
|
type DebtStatus string
|
|
|
|
const (
|
|
DebtObserved DebtStatus = "observed"
|
|
DebtEligible DebtStatus = "eligible"
|
|
DebtScheduled DebtStatus = "scheduled"
|
|
DebtRepaired DebtStatus = "repaired"
|
|
DebtWithdrawn DebtStatus = "withdrawn"
|
|
)
|
|
|
|
// ObservationKind names where one piece of evidence came from. It is the kind
|
|
// of the source fact, never an interpretation of it.
|
|
type ObservationKind string
|
|
|
|
const (
|
|
ObservationBlockReason ObservationKind = "block_reason"
|
|
ObservationFailureClass ObservationKind = "failure_class"
|
|
ObservationReviewFinding ObservationKind = "review_finding"
|
|
ObservationPlanMismatch ObservationKind = "plan_mismatch"
|
|
ObservationDeferredFinding ObservationKind = "deferred_finding"
|
|
ObservationManualIntervention ObservationKind = "manual_intervention"
|
|
ObservationWorkerFailure ObservationKind = "worker_observation"
|
|
)
|
|
|
|
// SignatureVersion prefixes every signature this build produces. Normalization
|
|
// rules will change, and without a version a change silently regroups every
|
|
// historical observation, moving the recurrence counts that eligibility was
|
|
// already decided on. A v2 signature never matches a v1 one, so old evidence
|
|
// keeps the grouping it was counted under.
|
|
const SignatureVersion = "v1"
|
|
|
|
// DebtSignature is the mechanical dedup key. Every part is a typed fact, never
|
|
// prose, because exact signature match is the only path that attaches evidence
|
|
// without a human. An empty part becomes "-" so the arity never varies.
|
|
func DebtSignature(class DebtClass, reason, scope, component string) string {
|
|
part := func(s string) string {
|
|
s = strings.TrimSpace(strings.ToLower(s))
|
|
s = strings.ReplaceAll(s, ":", "_")
|
|
if s == "" {
|
|
return "-"
|
|
}
|
|
return s
|
|
}
|
|
return strings.Join([]string{SignatureVersion, part(string(class)), part(reason), part(scope), part(component)}, ":")
|
|
}
|
|
|
|
// DebtComponent reduces a repository path to the unit that owns it. Two
|
|
// segments is the whole rule: it keeps internal/store distinct from
|
|
// internal/herdr without splitting one package across files.
|
|
func DebtComponent(path string) string {
|
|
path = strings.TrimSpace(strings.Trim(path, "/"))
|
|
if path == "" {
|
|
return ""
|
|
}
|
|
parts := strings.Split(path, "/")
|
|
if len(parts) == 1 {
|
|
return parts[0]
|
|
}
|
|
return parts[0] + "/" + parts[1]
|
|
}
|
|
|
|
// DebtObservation is one piece of counted evidence. Provenance is required and
|
|
// is either an event id or, for history imported from a markdown ledger that
|
|
// predates this projection, an explicit legacy reference. Exactly one, because
|
|
// a fabricated event id would break the rule the ledger exists to enforce.
|
|
type DebtObservation struct {
|
|
EventID string `json:"event_id,omitempty"`
|
|
LegacyRef string `json:"legacy_ref,omitempty"`
|
|
TaskID string `json:"task_id,omitempty"`
|
|
Kind ObservationKind `json:"kind"`
|
|
Signature string `json:"signature"`
|
|
Detail string `json:"detail,omitempty"`
|
|
Paths []string `json:"paths,omitempty"`
|
|
At time.Time `json:"at"`
|
|
// Repeats is how many times this one incident recurred. It is intensity,
|
|
// never recurrence: one worker stuck in a five-second retry loop produced
|
|
// 301 repeats of a single failure, and counting those as 301 pieces of
|
|
// evidence would make one broken worker look like chronic, system-wide
|
|
// debt. Recurrence is the number of independent observations.
|
|
Repeats int `json:"repeats,omitempty"`
|
|
}
|
|
|
|
func (o DebtObservation) Validate() error {
|
|
if (o.EventID == "") == (o.LegacyRef == "") {
|
|
return fmt.Errorf("observation needs exactly one of event_id and legacy_ref")
|
|
}
|
|
if o.Kind == "" {
|
|
return fmt.Errorf("observation kind required")
|
|
}
|
|
if !strings.HasPrefix(o.Signature, SignatureVersion+":") {
|
|
return fmt.Errorf("observation signature %q is not %s", o.Signature, SignatureVersion)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// DebtItem groups observations that share a signature. Every count the
|
|
// priority model needs is derived from Observations rather than stored, so a
|
|
// count can never drift from the log that justifies it.
|
|
type DebtItem struct {
|
|
ID string `json:"id"`
|
|
Class DebtClass `json:"class"`
|
|
Status DebtStatus `json:"status"`
|
|
Symptom string `json:"symptom"`
|
|
Consequence string `json:"consequence,omitempty"`
|
|
Severity review.Severity `json:"severity,omitempty"`
|
|
Signatures []string `json:"signatures"`
|
|
Paths []string `json:"paths,omitempty"`
|
|
Observations []DebtObservation `json:"observations"`
|
|
IntroducedIn string `json:"introduced_in,omitempty"`
|
|
RepairBoundary string `json:"repair_boundary,omitempty"`
|
|
RepairTask string `json:"repair_task,omitempty"`
|
|
RepairCommit string `json:"repair_commit,omitempty"`
|
|
}
|
|
|
|
// Recurrence is how many times this shape has been seen.
|
|
func (d DebtItem) Recurrence() int { return len(d.Observations) }
|
|
|
|
// AffectedTasks is how broadly the shape has spread. Distinct tasks, because
|
|
// one task failing ten times is weaker evidence than ten tasks failing once.
|
|
func (d DebtItem) AffectedTasks() int {
|
|
seen := map[string]bool{}
|
|
for _, o := range d.Observations {
|
|
if o.TaskID != "" {
|
|
seen[o.TaskID] = true
|
|
}
|
|
}
|
|
return len(seen)
|
|
}
|
|
|
|
// BlockedTasks counts distinct tasks this shape actually stopped.
|
|
func (d DebtItem) BlockedTasks() int {
|
|
seen := map[string]bool{}
|
|
for _, o := range d.Observations {
|
|
if o.Kind == ObservationBlockReason && o.TaskID != "" {
|
|
seen[o.TaskID] = true
|
|
}
|
|
}
|
|
return len(seen)
|
|
}
|
|
|
|
// ManualInterventions counts recorded operator repairs. It reads zero on any
|
|
// log written before that recording exists, which is why the ledger reports it
|
|
// as an evidence gap rather than as an absence of operator cost.
|
|
func (d DebtItem) ManualInterventions() int {
|
|
n := 0
|
|
for _, o := range d.Observations {
|
|
if o.Kind == ObservationManualIntervention {
|
|
n++
|
|
}
|
|
}
|
|
return n
|
|
}
|
|
|
|
// Components is the breadth of the shape across the tree.
|
|
func (d DebtItem) Components() []string {
|
|
seen := map[string]bool{}
|
|
out := []string{}
|
|
for _, o := range d.Observations {
|
|
for _, p := range o.Paths {
|
|
if c := DebtComponent(p); c != "" && !seen[c] {
|
|
seen[c] = true
|
|
out = append(out, c)
|
|
}
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
// Imported reports whether any evidence here came from a markdown ledger
|
|
// rather than from an event. A reader can then tell a counted fact from an
|
|
// imported claim without opening the observations.
|
|
func (d DebtItem) Imported() bool {
|
|
for _, o := range d.Observations {
|
|
if o.LegacyRef != "" {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// EvidenceGap is a debt signal the ledger knows it cannot see. Reporting the
|
|
// gap is the point: a ledger that silently omits what it cannot record reads
|
|
// as "no operator cost" when it means "operator cost is not recorded".
|
|
type EvidenceGap struct {
|
|
Kind ObservationKind `json:"kind"`
|
|
Reason string `json:"reason"`
|
|
// Durable is false when no event type carries this evidence at all. It is
|
|
// true when the log could carry it and this particular log does not.
|
|
Durable bool `json:"durable"`
|
|
}
|
|
|
|
// DebtLedger is the whole read model: what history establishes, and what it
|
|
// cannot.
|
|
type DebtLedger struct {
|
|
Items []DebtItem `json:"items"`
|
|
Gaps []EvidenceGap `json:"gaps"`
|
|
// Events is how many log entries the projection folded, so a caller can
|
|
// tell an empty ledger from an unread log.
|
|
Events int `json:"events"`
|
|
}
|
|
|
|
// DebtClassForBlockReason maps a typed block reason to a class. A reason that
|
|
// is a normal lifecycle stop rather than a cost returns false: waiting for a
|
|
// human decision is the system working, not debt.
|
|
func DebtClassForBlockReason(r BlockReason) (DebtClass, bool) {
|
|
switch r {
|
|
case BlockReasonLeaseFailure, BlockReasonWorkerOffline, BlockReasonLeaseExpired, BlockReasonSystem:
|
|
return DebtOperational, true
|
|
case BlockReasonHandoffValidation, BlockReasonPlanMismatch:
|
|
return DebtCorrectness, true
|
|
}
|
|
return "", false
|
|
}
|
|
|
|
// DebtClassForFailureClass maps a worker failure class to a class. Only the
|
|
// classes a worker actually emits are listed; an unknown one is not guessed at.
|
|
func DebtClassForFailureClass(f string) (DebtClass, bool) {
|
|
switch f {
|
|
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired", "handoff_unanswered":
|
|
return DebtOperational, true
|
|
case "invalid_handoff":
|
|
return DebtCorrectness, true
|
|
}
|
|
return "", false
|
|
}
|