Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 2d28f7b462 | |||
| a757cffc78 | |||
| 2f61a99986 | |||
| 595a1d3533 | |||
| 0797432d6f | |||
| 3c7cf95d8c | |||
| 0f83559ecc |
@@ -2274,3 +2274,256 @@ tmux -L orchestra kill-session -t <session>
|
||||
|
||||
Task `06G4JX6MSQEP7N0D5JWW9EP5X4` is the other run 12 task and is `in_review` on
|
||||
a pull request. It is real work and should be reviewed or failed, not cleaned.
|
||||
|
||||
### F18 closed: a bounded observation ring
|
||||
|
||||
`d6ee10f`. `WorkerHealth` now carries up to sixteen distinct observations, each
|
||||
with a repeat count and first/last times. Collapsing is by message, not by
|
||||
position, because a loop interleaved with other failures would otherwise still
|
||||
flush the ring. Eviction drops the least recently seen, so a loop keeps its
|
||||
slot but carries the count that says what it is. `last_error` and `error_at`
|
||||
keep their wire names and still report only the newest failure.
|
||||
|
||||
The ring lives in memory beside `last_error` and is not persisted, which is
|
||||
what `last_error` already did across a restart.
|
||||
|
||||
Deployed and verified on both halves. It reports nothing yet, because
|
||||
`omitempty` hides an empty ring and nothing has failed since the deploy. No UI
|
||||
renders it: `internal/ui` never showed `last_error` either, so
|
||||
`GET /v1/federation/workers` is the only reader today.
|
||||
|
||||
### Run 12 closed out
|
||||
|
||||
PR 17 merged, task `06G4JX6MSQEP7N0D5JWW9EP5X4` completed at v32, pane closed
|
||||
by the normal completion path. The quality gate was re-run independently
|
||||
against the submitted commit `419e17fc` before the merge, rather than trusted
|
||||
from the worker's own report:
|
||||
|
||||
```text
|
||||
bash -n scripts/*.sh syntax ok
|
||||
bash scripts/orchestra_e2e_healthcheck.sh OK - all healthchecks passed, exit 0
|
||||
bash scripts/test_healthcheck.sh all checks passed, exit 0
|
||||
```
|
||||
|
||||
Worker state is empty: no sessions, no leases, no release transactions, and
|
||||
`tmux -L orchestra ls` lists nothing. The two panes the previous section left
|
||||
for an operator were already gone by then.
|
||||
|
||||
| Half | Revision |
|
||||
|---|---|
|
||||
| Coordinator, homesrv container | `d6ee10f` |
|
||||
| Worker, workpc systemd | `d6ee10f` |
|
||||
|
||||
```text
|
||||
commit d6ee10f02843e2267c859b03d17db00995941e64
|
||||
coordinator sha256 f8a847477ae0ac03fe78338775f52ed1f167f0e67eab557feffe4d67da3ad99e
|
||||
worker sha256 fe7d2188b12369fb7682d3424fa11e2db47abed9c55313291a4b467b5972d02d
|
||||
```
|
||||
|
||||
### Still open after run 12
|
||||
|
||||
- **29 blocked `test-e2e` tasks**, mostly burn-in debris. Operator hygiene, not
|
||||
lifecycle code.
|
||||
- **Three queued `correx` tasks** that cannot be scheduled, because `correx` has
|
||||
no entry in the coordinator's `config.jsonc`. Either give it one or block them
|
||||
explicitly.
|
||||
- **Gitea returns 422 for a review on your own pull request**, so
|
||||
`REQUEST_CHANGES` needs a separate bot account for `ORCHESTRA_GITEA_TOKEN`.
|
||||
- **A blocked task that is never reopened can still loop** on a dead release
|
||||
transaction. F59 covers failed, not blocked.
|
||||
|
||||
## Run 13, 2026-08-28: F60, and the ring proven under a real failure
|
||||
|
||||
### F60: a transaction must settle deterministically, in every case
|
||||
|
||||
`3c7cf95`. F58 and F59 each fixed one case of the same rule. F60 states the
|
||||
rule and covers the rest:
|
||||
|
||||
> a transaction that can no longer be legitimately committed must be abandoned
|
||||
> deterministically, and must never spin on an answer that cannot change.
|
||||
|
||||
- **Terminal is failed or completed.** Both drop the transaction and free the
|
||||
session. Completed was the gap F59 left.
|
||||
- **Blocked keeps the transaction**, because a reopen returns the task to the
|
||||
queue and that exact owner can still commit.
|
||||
- **`TaskBlocked` now retains the ending epoch**, as `TaskReleased` already did.
|
||||
Without it a reopened task has no epoch for the late-handoff path to fence
|
||||
against, so keeping the transaction would be a lie.
|
||||
- **A refusal parks the commit**, 30s backing off to a 5 minute cap. Any event
|
||||
about the task un-parks it. A reopen arrives as `TaskCorrected`, so the rule
|
||||
cannot be a list of event types.
|
||||
- **A transport failure is not an answer** and retries at once. The park keys on
|
||||
a 4xx `StatusError`, never on any error.
|
||||
|
||||
### Proven live, and the contrast with run 10
|
||||
|
||||
Task `06G4KVSHAK9B8M9K8HENAF23CG`, same race-guard rig: force the expiry inside
|
||||
the push, then hand the task to `race-guard-probe`.
|
||||
|
||||
```text
|
||||
19:54:10.888 transaction 06G4KVYT91GQH5ZYY6CNG6FWZ8 opens at prepared
|
||||
19:54:10.907 forced expiry accepted
|
||||
19:54:10.916 successor lease to race-guard-probe accepted
|
||||
19:54:11.776 commit refused once, 409 lease not owned, parked
|
||||
19:54:15.741 superseded by lease 06G4KVYSXJH053K3DAG6FKHD20, transaction abandoned
|
||||
```
|
||||
|
||||
Run 10's equivalent ran roughly 5,000 retries over seven hours and needed an
|
||||
operator to clear the state file by hand. This one asked once and settled
|
||||
itself in four seconds. Worker state afterwards: no sessions, no leases, no
|
||||
transactions, and `tmux -L orchestra ls` empty.
|
||||
|
||||
### F18 populated under a real failure
|
||||
|
||||
The gap the previous section recorded is closed. The same run produced this
|
||||
health payload, two distinct observations rather than one overwritten slot:
|
||||
|
||||
```json
|
||||
"observations": [
|
||||
{"message": "release 06G4KVSHAK9B8M9K8HENAF23CG commit: federation: 409 Conflict: lease not owned",
|
||||
"count": 1, "first": "2026-08-28T19:54:11.776815524Z", "last": "2026-08-28T19:54:11.776815524Z"},
|
||||
{"message": "release 06G4KVSHAK9B8M9K8HENAF23CG superseded by lease 06G4KVYSXJH053K3DAG6FKHD20: abandoning transaction 06G4KVYT91GQH5ZYY6CNG6FWZ8",
|
||||
"count": 1, "first": "2026-08-28T19:54:15.74116014Z", "last": "2026-08-28T19:54:15.74116014Z"}
|
||||
]
|
||||
```
|
||||
|
||||
Under the old single slot the abandonment message would have erased the 409
|
||||
that caused it, and the causal chain would have been unreadable. Both counts
|
||||
are 1, which is itself the F60 evidence: nothing looped.
|
||||
|
||||
### Deployed state
|
||||
|
||||
| Half | Revision |
|
||||
|---|---|
|
||||
| Coordinator, homesrv container | `3c7cf95` |
|
||||
| Worker, workpc systemd | `3c7cf95` |
|
||||
|
||||
```text
|
||||
commit 3c7cf95d8cdaaff5dcec460c5d7874f7f4f0dbea
|
||||
coordinator sha256 3fec1e99a638da851f5c970abaf80a006dc87425936f71fe33f10830f4bac573
|
||||
worker sha256 a373445f167f0a321dfc3866a1d2ed703b670b0fb3c188a8d00ad8099586dfdf
|
||||
```
|
||||
|
||||
`orchestra-f18-baseline` tags `d6ee10f`, the post-F18 deployed baseline, and is
|
||||
pushed. The branch is pushed too: 91 commits, because nothing had gone up since
|
||||
`97a9c65` on 2026-07-31.
|
||||
|
||||
### Still open
|
||||
|
||||
Unchanged and all operational rather than runtime: 29 blocked `test-e2e` tasks,
|
||||
three unschedulable `correx` tasks, and the Gitea 422 on reviewing your own
|
||||
pull request. The rig task above will requeue when the probe lease expires and
|
||||
start fresh, which is correct: its anchor was abandoned, not committed.
|
||||
|
||||
## Run 14, 2026-08-28: plan-phase execution proven on `3c7cf95`
|
||||
|
||||
Four tasks. Two carried a full lifecycle to a pull request, one failed for a
|
||||
cause outside Orchestra, and the F60 rig task failed on its retry budget.
|
||||
|
||||
### The rung: the worker runs the sealed plan's commands
|
||||
|
||||
Task `06G4M8WHGQ4P3GQMPEEH0RJRHM`, plan `789ed6a8477b`, both phases verified
|
||||
against the plan document rather than the request.
|
||||
|
||||
```text
|
||||
plan phase-1 - run: ["bash", "-n", "scripts/orchestra_e2e_healthcheck.sh"]
|
||||
- run: ["bash", "scripts/orchestra_e2e_healthcheck.sh"]
|
||||
event v19 commands identical, exit_codes [0,0], at_sha 3b66b2b6a669,
|
||||
evidence_ref a5dbc4ecc9ce, status awaiting_manual_verification
|
||||
|
||||
plan phase-2 - run: ["bash", "-n", "scripts/orchestra_e2e_healthcheck.sh"]
|
||||
- run: ["bash", "-n", "scripts/test_healthcheck.sh"]
|
||||
- run: ["bash", "scripts/orchestra_e2e_healthcheck.sh"]
|
||||
event v20 commands identical, exit_codes [0,0,0], same at_sha and plan_ref
|
||||
```
|
||||
|
||||
Every record binds `plan_ref`, `phase_id`, `at_sha`, `evidence_ref`,
|
||||
`lease_epoch` and `harness_id`. Rotation ran between every work phase with
|
||||
`TaskPickupValidated` each time.
|
||||
|
||||
**Not yet proven:** a successor picking up mid-implement. Both phases verified
|
||||
under one lease epoch, so no rotation separated phase 1 from phase 2.
|
||||
|
||||
**The planner found the legal route to a forbidden command.**
|
||||
`scripts/test_healthcheck.sh` may not be executed, but `["bash", "-n", ...]`
|
||||
covers any path, so it syntax-checked the file it could not run.
|
||||
|
||||
### The refusal loop is actionable, not merely observable
|
||||
|
||||
Task `06G4M6HF1Z3EREX1X3NEKSHP24` sealed a plan naming
|
||||
`bash scripts/test_healthcheck.sh`, which the policy forbids.
|
||||
|
||||
```text
|
||||
20:45:02 phase request refused: "bash scripts/test_healthcheck.sh" is not in the policy
|
||||
20:45:33 implement <- the planner rewrote and resealed, 31 seconds later
|
||||
20:48:03 in_review
|
||||
```
|
||||
|
||||
F52 refused on the planner, F55's answer path delivered the reason, and the
|
||||
agent corrected. Count stayed at 1 for both tasks that hit it. A brief that
|
||||
states the policy avoids the round trip entirely: the `--list-checks` task was
|
||||
told the allowed argv and sealed on the first attempt.
|
||||
|
||||
### F61, recorded and deliberately not built
|
||||
|
||||
The verification allow-list is enforced and never stated to the planner. Every
|
||||
plan that guesses pays one refused round trip. It is not a one-liner:
|
||||
`agentctx.Input.Policy` is filled from the worker's `SafeOperations`, while the
|
||||
verification policy is coordinator-side by design, so this needs a new channel.
|
||||
|
||||
The reason to promote it later is local models. A strong planner consumed the
|
||||
refusal and repaired the plan in 31 seconds. A weaker one may propose forbidden
|
||||
commands repeatedly, because it cannot infer the allowed substitute. Then F61
|
||||
stops being latency and becomes model compatibility.
|
||||
|
||||
### opencode is not running, and the reason was only in the pane
|
||||
|
||||
Task `06G4M6HH5BAM235AC4PWYXF1HM` failed at attempt 3 having never left
|
||||
`frame`. Its worktree held only `launch.md`. The pane said why. The blocked
|
||||
thing is the selected model, not the OpenCode Zen free tier, which works:
|
||||
|
||||
```text
|
||||
This model is not available in your country.
|
||||
▣ Build · Muse Spark 1.2 Free OpenCode Zen
|
||||
```
|
||||
|
||||
Orchestra behaved correctly throughout: lease, no progress, expire, requeue,
|
||||
fail. Two things are worth keeping.
|
||||
|
||||
- **The adapter cannot resolve a session file for opencode**, so activity is
|
||||
`unknown` and the worker falls back to agent status. The failure therefore
|
||||
reads as "done and pane unchanged" rather than "dead". Orchestra cannot tell
|
||||
finished from never-started on this harness.
|
||||
- **No observation carried the cause.** Four of them named the lease, the
|
||||
rotation and the activity. The fatal line existed only in the pane capture,
|
||||
which Orchestra publishes but nothing summarises. Parsing harness chrome is
|
||||
not Orchestra's job; surfacing the capture is.
|
||||
|
||||
**Do not route to `workpc-opencode` until its model is reachable.** Every task
|
||||
sent there burns three attempts and then fails. `~/.config/opencode/opencode.jsonc`
|
||||
sets no top-level `model`, so OpenCode picks its own default. The file already
|
||||
declares reachable `nvidia-nim` and `llama-cpp` providers.
|
||||
|
||||
### The retry budget counts idleness, not just failure
|
||||
|
||||
The F60 rig task `06G4KVSHAK9B8M9K8HENAF23CG` failed at attempt 3 mid-review.
|
||||
Two attempts were rig-induced expiries and the third was a natural idle
|
||||
expiry. Every lease expiry increments `attempt`, and an agent going quiet at a
|
||||
phase boundary is a routine expiry here. A task making real progress can spend
|
||||
its retry budget on idleness. The two clean tasks never expired at all, so this
|
||||
is a dynamic to watch rather than a defect to fix.
|
||||
|
||||
F59 was confirmed live twice more: both failed tasks left no release
|
||||
transaction and no session behind, on either worker.
|
||||
|
||||
### Result
|
||||
|
||||
| Task | Harness | Outcome |
|
||||
|---|---|---|
|
||||
| `06G4M6HF1Z3EREX1X3NEKSHP24` | workpc-claude | in_review, recovered from a policy refusal |
|
||||
| `06G4M8WHGQ4P3GQMPEEH0RJRHM` | workpc-claude | in_review, plan-phase execution proven |
|
||||
| `06G4M6HH5BAM235AC4PWYXF1HM` | workpc-opencode | failed, model unavailable |
|
||||
| `06G4KVSHAK9B8M9K8HENAF23CG` | workpc-claude | failed, retry budget |
|
||||
|
||||
Both workers hold no sessions and no release transactions. Deployed pair is
|
||||
still `3c7cf95` on both halves.
|
||||
|
||||
+410
@@ -0,0 +1,410 @@
|
||||
# Debt ledger: design from what the tree already supports
|
||||
|
||||
Written 2026-08-29 against `3c7cf95`. Read with `BURNIN.md` (the run ledger),
|
||||
`PLAN-SPEC-DESIGN.md` and `AUDIT.md`.
|
||||
|
||||
This is a design, not an implementation. Nothing here has been built.
|
||||
|
||||
## The problem being solved
|
||||
|
||||
AI raises implementation throughput faster than it raises integration, cleanup,
|
||||
observability, consolidation and architectural understanding. The residue is
|
||||
fixes, compatibility paths, operational workarounds, duplicate config, adapter
|
||||
gaps and one-off patches. Some are justified. Some become permanent because
|
||||
nobody remembers why they were added or what they have cost since.
|
||||
|
||||
The target is a first-class debt ledger driven by evidence from real tasks, not
|
||||
a second TODO list.
|
||||
|
||||
## Classes
|
||||
|
||||
Four, and no more. A fifth class invented at runtime makes the priority model
|
||||
meaningless.
|
||||
|
||||
| Class | Meaning |
|
||||
|---|---|
|
||||
| correctness | Known behaviour is wrong or violates an invariant. |
|
||||
| operational | The system works, but diagnosis, recovery, deployment, observability or operation repeatedly costs time. |
|
||||
| structural | Duplication or architecture is demonstrably raising the cost of future changes. |
|
||||
| polish | Cleanup or consistency work with no demonstrated cost yet. |
|
||||
|
||||
Correctness and operational debt gain priority quickly. Structural debt needs
|
||||
evidence that it is causing repeated work. Polish never becomes work on its own.
|
||||
|
||||
## 1. What Orchestra already has
|
||||
|
||||
The event log is the durable spine, and it already carries most of what a debt
|
||||
ledger needs. Every row below is mechanically countable today, with no new
|
||||
instrumentation.
|
||||
|
||||
| Evidence | Source | Carries |
|
||||
|---|---|---|
|
||||
| Failure shape | `TaskFailed`, `TaskBlocked` | 12 typed `block_reason` values, 6 `failure_class` values |
|
||||
| Retry cost | `Task.Attempt`, `NextRetryAt`, `FailureClass` | how many leases a task burned, durable in the projection |
|
||||
| Review findings | `ReviewRecorded` | `{ID, Severity, File, Line, Claim, Evidence}`, bound to a `ResultSHA` |
|
||||
| Out-of-scope discoveries | `DeferredFindingRecorded` | `{Summary, Why}`, already recorded outside agent context |
|
||||
| Plan contradicted by code | `PlanMismatchRecorded` | `{PlanRef, PhaseID, AtSHA, Observed, Contradicts, Evidence[]}` |
|
||||
| Verification history | `PlanPhaseVerified` | `{Commands, ExitCodes, AtSHA, EvidenceRef}` per phase |
|
||||
| Rework rounds | `TaskChangesRequested`, `TaskSubmitted` | how many times a change went back |
|
||||
| Provenance | `TaskCreated` | `Source`, `ExternalID`, `Parent` |
|
||||
|
||||
Three existing patterns matter more than the data.
|
||||
|
||||
**Global-subject events already work.** `QuotaReported` and `StandupAdvisory`
|
||||
use `TaskID: "system"` and are whitelisted in `internal/domain/domain.go:279`.
|
||||
A debt item needs no new subject mechanism.
|
||||
|
||||
**`GenerateStandupAdvisory` is the precedent for the shape.** It is a
|
||||
persisted, read-only recommendation, and applying it is a separate
|
||||
approval-gated operation. A debt ledger is that pattern with better inputs.
|
||||
|
||||
**`CheckSubmission` is the precedent for eligibility.** It is a pure function
|
||||
of task, commit and gate run, returning `{Eligible bool, Reasons []string}`.
|
||||
Debt eligibility should have the same shape.
|
||||
|
||||
## 2. What evidence is missing
|
||||
|
||||
**Worker observations are not durable.** This is the largest gap. The F18 ring
|
||||
lives in worker memory and reaches the coordinator inside `WorkerHealth` on
|
||||
heartbeat. `Registry.persistedState` holds captures, commands and workers only,
|
||||
with no health. A coordinator restart erases every observation.
|
||||
|
||||
Operational debt is exactly what that ring holds. Repeated 409s, unrenewed
|
||||
leases, adapter gaps.
|
||||
|
||||
**No operator-intervention record.** Every manual repair in run 12 and run 13
|
||||
left no trace in Orchestra. State-file edits, worker restarts, two manual
|
||||
transaction cleanups. "Required manual recovery" is the strongest priority
|
||||
signal available, and it is currently unrecorded.
|
||||
|
||||
**No diagnosis-cost signal.** Time spent diagnosing is not measured. The
|
||||
closest proxy is wall time in `blocked` or `needs_attention`.
|
||||
|
||||
**No component dimension.** Findings carry file paths. Tasks carry none.
|
||||
Breadth across components has to be derived from paths.
|
||||
|
||||
**No link from a repair commit back to what it repaired.** The Fxx-to-commit
|
||||
mapping exists only in `BURNIN.md` prose.
|
||||
|
||||
**Deferred findings are too thin.** `{Summary, Why}` has no class, no severity,
|
||||
no paths and no evidence refs.
|
||||
|
||||
## 3. Minimal durable data model
|
||||
|
||||
No new subsystem. Two event types on the existing spine, one projection beside
|
||||
`Task`.
|
||||
|
||||
```go
|
||||
// DebtObservation is one piece of counted evidence, always pointing at the
|
||||
// event that produced it. Provenance is an event id, never prose.
|
||||
type DebtObservation struct {
|
||||
// Exactly one of EventID and LegacyRef. See section 8.
|
||||
EventID string `json:"event_id,omitempty"`
|
||||
LegacyRef string `json:"legacy_ref,omitempty"`
|
||||
TaskID string `json:"task_id,omitempty"`
|
||||
Kind string `json:"kind"` // block_reason, failure_class, review_finding,
|
||||
// plan_mismatch, manual_intervention, deferred_finding
|
||||
Signature string `json:"signature"` // the mechanical dedup key, see section 4
|
||||
Detail string `json:"detail"`
|
||||
Paths []string `json:"paths,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
type DebtItem struct {
|
||||
ID string `json:"id"` // continues the Fxx namespace
|
||||
Class DebtClass `json:"class"` // correctness, operational, structural, polish
|
||||
Status DebtStatus `json:"status"` // observed, eligible, scheduled, repaired, withdrawn
|
||||
Symptom string `json:"symptom"`
|
||||
Consequence string `json:"consequence"`
|
||||
Severity Severity `json:"severity"` // reuse review.Severity
|
||||
Paths []string `json:"paths,omitempty"`
|
||||
Signatures []string `json:"signatures"` // every key that attaches here
|
||||
Observations []DebtObservation `json:"observations"`
|
||||
IntroducedIn string `json:"introduced_in,omitempty"` // task id or commit
|
||||
RepairBoundary string `json:"repair_boundary,omitempty"`
|
||||
RepairTask string `json:"repair_task,omitempty"`
|
||||
RepairCommit string `json:"repair_commit,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
Every counted field the priority model needs is derived, never stored:
|
||||
|
||||
```text
|
||||
recurrence len(Observations)
|
||||
blocked_tasks distinct TaskID where Kind is block_reason
|
||||
manual_interventions count of Kind == manual_intervention
|
||||
breadth distinct components derived from Paths
|
||||
```
|
||||
|
||||
A stored count invites drift from the log. A derived one cannot drift.
|
||||
|
||||
Two events carry it. `DebtObserved` appends one observation. `DebtItemUpdated`
|
||||
records class, status, severity or a merge. Both need adding to the `allowed`
|
||||
map and a validator, which is the work every existing event type already did.
|
||||
|
||||
## 4. Deduplication without corrupting provenance
|
||||
|
||||
**A signature is computed from typed facts, never from prose.**
|
||||
|
||||
```text
|
||||
operational class + block_reason + harness + normalized component
|
||||
correctness class + failure_class + normalized component
|
||||
structural class + finding severity + normalized component
|
||||
```
|
||||
|
||||
Exact signature match attaches automatically. That is the only automatic path.
|
||||
|
||||
**Signatures are versioned, in the string itself.**
|
||||
|
||||
```text
|
||||
v1:operational:lease_not_renewed:workpc-opencode:federation
|
||||
```
|
||||
|
||||
Normalization rules will change. Without a version, changing them silently
|
||||
regroups every historical observation, and the recurrence counts that drive
|
||||
eligibility move underneath the items that already used them. A `v2` signature
|
||||
never matches a `v1` one, so old evidence keeps the grouping it was counted
|
||||
under.
|
||||
|
||||
**A model may propose, never merge.** Clustering emits `DebtMergeSuggested`
|
||||
carrying both item ids and its reasoning. The merge is an operator action or a
|
||||
policy threshold, recorded as `DebtItemUpdated`.
|
||||
|
||||
**Merges are additive.** The surviving item gains the other's signatures and
|
||||
observations. The absorbed item becomes `withdrawn` with a pointer, and its
|
||||
observations keep their original event ids. Nothing is rewritten, so a bad
|
||||
merge is reversible by reading the log.
|
||||
|
||||
**Every observation names its event.** A debt item's evidence is always
|
||||
checkable against the log that produced it. That rule is what stops a fuzzy
|
||||
proposal from becoming an unverifiable claim.
|
||||
|
||||
## 5. Eligibility policy
|
||||
|
||||
A pure function, mirroring `CheckSubmission`.
|
||||
|
||||
```go
|
||||
func CheckDebtEligibility(item DebtItem) DebtCheck // {Eligible bool, Reasons []string}
|
||||
```
|
||||
|
||||
| Class | Becomes eligible when |
|
||||
|---|---|
|
||||
| correctness | first confirmed observation, always |
|
||||
| operational | recurrence >= 3 across >= 2 distinct tasks, or >= 1 manual intervention |
|
||||
| structural | >= 3 review findings, or >= 2 tasks blocked or reworked in the same component |
|
||||
| polish | never automatically, operator promotion only |
|
||||
|
||||
`Reasons` lists what fired and what did not. "Not eligible" alone sends an
|
||||
operator reading code, which is the mistake `SubmissionCheck` already documents.
|
||||
|
||||
Eligible means a task may be created. It does not create one.
|
||||
|
||||
## 6. How maintenance and consolidation fit the existing lifecycle
|
||||
|
||||
A maintenance task is an ordinary task. Source `debt`, external id the debt item
|
||||
id, phases as usual.
|
||||
|
||||
```text
|
||||
research → plan → implement → verification → review → submission
|
||||
```
|
||||
|
||||
Nothing in the lifecycle changes. One extension is needed: the task's brief must
|
||||
carry the debt item's evidence. That is the mechanism research and plan
|
||||
artifacts already use, a CAS ref on the task.
|
||||
|
||||
A consolidation task is the same thing with a research brief that asks:
|
||||
|
||||
- which temporary paths are still necessary?
|
||||
- which fixes now duplicate each other?
|
||||
- which compatibility branches are obsolete?
|
||||
- which config or state is represented twice?
|
||||
- which abstractions exist only because of defects that have since disappeared?
|
||||
- what can now be deleted safely?
|
||||
|
||||
Its first output is a deletion plan, never an automatic refactor. Verification
|
||||
uses the project's existing policy. Review is the normal independent review.
|
||||
|
||||
**The repair gets no discount.** Classification as debt changes what gets
|
||||
scheduled, never what gets checked.
|
||||
|
||||
### Success metrics for maintenance work
|
||||
|
||||
Lines of code are not a metric. Prefer evidence that can be checked:
|
||||
|
||||
- behaviour preserved
|
||||
- tests preserved or strengthened
|
||||
- branches removed
|
||||
- obsolete types removed
|
||||
- compatibility code removed
|
||||
- duplicate config removed
|
||||
- manual recovery paths eliminated
|
||||
|
||||
## 7. What must not be automated
|
||||
|
||||
- No autonomous cleanup agent, and no repo-wide sweep.
|
||||
- No LLM score treated as authoritative. Counted facts decide, models propose.
|
||||
- No automatic merge of debt items.
|
||||
- No maintenance task from polish without an operator.
|
||||
- No deletion without an independent review.
|
||||
- No auto-closing an item because a commit touched the file. Closure needs a
|
||||
repair task or a stated run of clean evidence.
|
||||
- No new classes invented at runtime.
|
||||
|
||||
## 8. Migration, so there is one track and not two
|
||||
|
||||
`BURNIN.md` holds 58 Fxx entries in the form `| F52 | commit | prose |`.
|
||||
`AUDIT.md` holds the older narrative.
|
||||
|
||||
One-time import, by hand, with a class assigned per row. Each becomes a
|
||||
`DebtItem` with `Status: repaired`, its commit as `RepairCommit`, and one
|
||||
observation citing the ledger.
|
||||
|
||||
**Imported observations carry legacy provenance, never a fabricated event id.**
|
||||
The Fxx history predates the capability that would have produced an event, and
|
||||
inventing one would break the provenance rule the ledger exists to enforce.
|
||||
|
||||
```go
|
||||
EventID string `json:"event_id,omitempty"`
|
||||
LegacyRef string `json:"legacy_ref,omitempty"` // "BURNIN.md:F18"
|
||||
```
|
||||
|
||||
Exactly one is required. A reader can then tell a counted fact from an imported
|
||||
claim at a glance, and the counts that drive eligibility can exclude imported
|
||||
evidence if that turns out to matter.
|
||||
|
||||
**Keep the namespace.** New items continue at F62. Two numbering schemes would
|
||||
be the first structural debt the ledger itself creates.
|
||||
|
||||
After import, `BURNIN.md` stops being the item ledger and remains the run
|
||||
narrative. The defect table becomes generated output from the projection.
|
||||
|
||||
### The history that tests the model
|
||||
|
||||
These are not special cases. They are the check on whether the model can
|
||||
represent the history this project already has.
|
||||
|
||||
| History | Expected class | Why |
|
||||
|---|---|---|
|
||||
| F18, the single `last_error` slot | operational | Repeatedly destroyed causal evidence before it was fixed. |
|
||||
| F43 to F46, the submission path | correctness | Apparently implemented, actually unreachable. |
|
||||
| F61, planner learns policy by refusal | operational | Costs a round trip per plan, and becomes model compatibility on weaker planners. |
|
||||
| Duplicated `quality_gate` config | structural | Only if it keeps causing drift or operator mistakes. |
|
||||
| Old blocked burn-in tasks | neither | Hygiene, unless one exposes a runtime defect. |
|
||||
| The blocked release loop, F57 to F60 | correctness and operational | Wrong behaviour, and it required manual cleanup twice. |
|
||||
|
||||
## 9. Smallest slice to live-prove first
|
||||
|
||||
**A read-only projection over the existing log. No new events, no writes, no
|
||||
schema commitment.**
|
||||
|
||||
```text
|
||||
internal/domain/debt.go DebtItem, DebtObservation, signature, classification
|
||||
internal/store/debt_projection.go the fold over s.Events(0), and the gap report
|
||||
internal/operations/debt.go CheckDebtEligibility, and later the actions
|
||||
GET /v1/debt read-only, tui surface
|
||||
```
|
||||
|
||||
The projected type stays out of `operations`. Baking a read model into the
|
||||
command layer in the first slice is the mistake that would be hardest to undo
|
||||
later. `operations` owns actions: `CheckDebtEligibility` now, and
|
||||
`SuggestDebtMerge`, `PromoteDebtItem` and `ScheduleDebtRepair` when they exist.
|
||||
|
||||
The proof is what the projection can and cannot recover from canonical history.
|
||||
Section 2 already says worker observations are not durable and operator
|
||||
interventions are unrecorded, so demanding their reconstruction would be asking
|
||||
the projection to invent evidence.
|
||||
|
||||
The first burn-in is therefore three requirements, not one:
|
||||
|
||||
1. Recover every debt signal that canonical history actually encodes.
|
||||
2. Report, explicitly and per kind, where known debt cannot be reconstructed.
|
||||
3. Never infer a missing observation from `BURNIN.md`.
|
||||
|
||||
Incompleteness is part of the result, not a failure. The 409 loop lived in the
|
||||
F18 worker ring rather than in an event, so the ledger should say it has no
|
||||
durable evidence for that shape. That statement is what makes slice two
|
||||
necessary, measurably rather than by assertion.
|
||||
|
||||
What the projection should recover from the log alone:
|
||||
|
||||
- repeated `lease_expired` and `retry_limit` on one harness, which is the
|
||||
opencode failure shape across four tasks
|
||||
- the retry-idleness dynamic, attached to the tasks that expired
|
||||
- review findings grouped by component, if any repeat
|
||||
|
||||
A model that cannot represent debt already known is wrong, and no schema has
|
||||
been committed to yet. That is the cheapest place to find out.
|
||||
|
||||
**Slice two** makes worker observations durable, because that is the input the
|
||||
projection will most visibly lack.
|
||||
|
||||
**Slice three** adds `DebtObserved` and manual-intervention recording.
|
||||
|
||||
## Scope boundary
|
||||
|
||||
This design changes no part of the task lifecycle, phases, leases, review,
|
||||
submission, federation or the plan machinery. The only extensions it needs are
|
||||
two event types, one projection, one pure eligibility function and one
|
||||
read-only endpoint.
|
||||
|
||||
---
|
||||
|
||||
# Slice one, run against real history
|
||||
|
||||
Built and deployed as `a757cff`. `GET /v1/debt` folded 881 events from the
|
||||
live log, wrote nothing, and produced 15 candidate items and 3 gaps.
|
||||
|
||||
## What it recovered
|
||||
|
||||
```text
|
||||
v1:operational:lease_expired:workpc-opencode:lease r=41 tasks=4
|
||||
v1:operational:lease_expired:workpc-claude:lease r=29 tasks=13
|
||||
v1:operational:lease_failure:-:lease r=20 tasks=16
|
||||
v1:operational:system_error:-:lease r=11 tasks=7
|
||||
v1:correctness:handoff_validation:-:lease r=5 tasks=5
|
||||
v1:correctness:plan_mismatch:-:... r=1 tasks=1
|
||||
v1:structural:minor:-:scripts/orchestra_e2e_healthcheck.sh r=1 tasks=1
|
||||
```
|
||||
|
||||
The opencode failure shape is the top item, found mechanically. Run 14
|
||||
diagnosed the same thing by hand from a pane capture. The retry-idleness
|
||||
dynamic is the second item, and it is now a number: 29 expiries across 13
|
||||
tasks on one harness.
|
||||
|
||||
Seven items are eligible under the stated policy. Polish and the single
|
||||
structural finding correctly are not.
|
||||
|
||||
## What it reported that it could not see
|
||||
|
||||
```text
|
||||
durable=false manual_intervention no event type records an operator repair
|
||||
durable=false worker_observation worker health is not persisted
|
||||
durable=true deferred_finding carried by the log, this history has none
|
||||
```
|
||||
|
||||
The 409 release loop does not appear, and it should not. That evidence lived in
|
||||
the F18 worker ring, which no event carries. The ledger says so rather than
|
||||
inferring it, which is the result slice two exists to change.
|
||||
|
||||
## Four defects the first run exposed
|
||||
|
||||
**The component part is too coarse for lease evidence.** Everything
|
||||
lease-related normalizes to `lease`, so `lease_expired:workpc-claude` is one
|
||||
bucket holding idle agents, rig interference and real failures. Recurrence 29
|
||||
is true and the item is not a defect.
|
||||
|
||||
**`harness` is often empty on block reasons.** `TaskBlocked` payloads do not
|
||||
always carry `harness_id`, so `lease_failure:-` mixes harnesses that should be
|
||||
separate items.
|
||||
|
||||
**Path normalization mangled a mismatch reference.**
|
||||
`scripts/orchestra_e2e_healthcheck.sh:12` became
|
||||
`scripts/orchestra_e2e_healthcheck.sh_12`, because the signature sanitizer
|
||||
replaces `:` and the mismatch evidence field holds prose, not clean paths.
|
||||
|
||||
**Recurrence alone is the wrong sort.** 41 occurrences on 4 tasks currently
|
||||
outranks 29 on 13 tasks. Breadth is in the design and not yet in the ordering,
|
||||
which is the priority function slice one deliberately omitted.
|
||||
|
||||
None of these required a schema commitment to discover. That was the point of
|
||||
making the first slice read-only.
|
||||
@@ -157,6 +157,22 @@ type releaseTransaction struct {
|
||||
AgentReleased bool `json:"agent_released,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
// NextAttemptAt parks a commit the coordinator has refused. A refusal is
|
||||
// an answer about the task, not a transport failure, so it stays true
|
||||
// until something about the task changes.
|
||||
NextAttemptAt time.Time `json:"next_attempt_at,omitempty"`
|
||||
Attempts int `json:"attempts,omitempty"`
|
||||
}
|
||||
|
||||
// releaseBackoff spaces out refused commits. The first wait is long enough
|
||||
// that a parked transaction stops filling the observation ring, and the cap
|
||||
// keeps a reopen from waiting more than five minutes to be noticed.
|
||||
func releaseBackoff(attempts int) time.Duration {
|
||||
d := 30 * time.Second << (attempts - 1)
|
||||
if attempts < 1 || d > 5*time.Minute {
|
||||
return 5 * time.Minute
|
||||
}
|
||||
return d
|
||||
}
|
||||
type projectConfig struct {
|
||||
Repo string `json:"repo"`
|
||||
@@ -894,18 +910,31 @@ func (w *worker) advanceRelease(ctx context.Context, id string, s herdr.Session)
|
||||
w.releases[id] = tx
|
||||
_ = w.save()
|
||||
}
|
||||
if tx.Phase == "anchor_pushed" && time.Now().Before(tx.NextAttemptAt) {
|
||||
return
|
||||
}
|
||||
if tx.Phase == "anchor_pushed" {
|
||||
// The epoch comes from the transaction, not from w.leases: an expiry
|
||||
// replay deletes the lease, and the coordinator needs the epoch of the
|
||||
// lease this anchor was pushed under to accept the late commit.
|
||||
if err := w.api.Release(ctx, id, tx.Ref, tx.AnchorSHA, tx.ID, tx.LeaseEpoch, tx.LeaseVersion, w.sessionEvidence(ctx, id, s)); err != nil {
|
||||
tx.LastError, tx.UpdatedAt = err.Error(), time.Now().UTC()
|
||||
// A refusal is the coordinator's answer about who owns the task.
|
||||
// It cannot change until an event about that task does, so asking
|
||||
// again every five seconds only burns the observation ring. A
|
||||
// transport failure is the opposite and must retry at once.
|
||||
var refused *federation.StatusError
|
||||
if errors.As(err, &refused) && refused.Code >= 400 && refused.Code < 500 {
|
||||
tx.Attempts++
|
||||
tx.NextAttemptAt = time.Now().UTC().Add(releaseBackoff(tx.Attempts))
|
||||
}
|
||||
w.releases[id] = tx
|
||||
_ = w.save()
|
||||
w.recordError(fmt.Errorf("release %s commit: %w", id, err))
|
||||
return
|
||||
}
|
||||
tx.Phase, tx.LastError, tx.UpdatedAt = "event_committed", "", time.Now().UTC()
|
||||
tx.NextAttemptAt, tx.Attempts = time.Time{}, 0
|
||||
w.releases[id] = tx
|
||||
_ = w.save()
|
||||
}
|
||||
@@ -1313,6 +1342,13 @@ func (w *worker) once(ctx context.Context) error {
|
||||
if t, ok := created(e); ok {
|
||||
w.tasks[t.ID] = t
|
||||
}
|
||||
// Any event about this task is the change a parked commit was waiting
|
||||
// for. A reopen arrives as TaskCorrected, so this cannot be a list of
|
||||
// specific types without going stale.
|
||||
if tx, parked := w.releases[e.TaskID]; parked && !tx.NextAttemptAt.IsZero() {
|
||||
tx.NextAttemptAt, tx.Attempts = time.Time{}, 0
|
||||
w.releases[e.TaskID] = tx
|
||||
}
|
||||
if e.Type == "TaskLeased" {
|
||||
var p struct {
|
||||
HarnessID string `json:"harness_id"`
|
||||
@@ -1403,7 +1439,7 @@ func (w *worker) once(ctx context.Context) error {
|
||||
w.leases[e.TaskID] = l
|
||||
}
|
||||
}
|
||||
if e.Type == "TaskReleased" || e.Type == "TaskFailed" || e.Type == "TaskBlocked" {
|
||||
if e.Type == "TaskReleased" || e.Type == "TaskFailed" || e.Type == "TaskBlocked" || e.Type == "TaskCompleted" {
|
||||
if e.Type == "TaskReleased" {
|
||||
var p struct {
|
||||
TransactionID string `json:"transaction_id"`
|
||||
@@ -1425,12 +1461,13 @@ func (w *worker) once(ctx context.Context) error {
|
||||
// mapping protects nothing. F30: a transaction stuck at "prepared"
|
||||
// held the session forever once its pane was gone, health() kept
|
||||
// reporting ActiveTask, and the harness never leased again.
|
||||
// A failed task is terminal: no successor will ever lease it, so
|
||||
// its anchor protects nothing and its transaction can only retry
|
||||
// a refusal forever. Blocked is different, because a reopen still
|
||||
// produces a successor.
|
||||
// Failed and completed are terminal: no successor will ever lease
|
||||
// the task, so the anchor protects nothing and the transaction can
|
||||
// only retry a refusal forever. Blocked is different, because a
|
||||
// reopen returns the task to the queue and the epoch that ended is
|
||||
// still on record, so that exact commit can still be accepted.
|
||||
tx, releasing := w.releases[e.TaskID]
|
||||
if !releasing || tx.Ref == "" || e.Type == "TaskFailed" {
|
||||
if !releasing || tx.Ref == "" || e.Type == "TaskFailed" || e.Type == "TaskCompleted" {
|
||||
if releasing {
|
||||
delete(w.releases, e.TaskID)
|
||||
}
|
||||
|
||||
@@ -1327,3 +1327,120 @@ func TestObservationRingEvictsLeastRecentlySeen(t *testing.T) {
|
||||
t.Fatal("least recently seen entry survived")
|
||||
}
|
||||
}
|
||||
|
||||
// F60. A refusal is an answer about the task, not a transport failure, and it
|
||||
// stays true until something about that task changes. Run 10's blocked task
|
||||
// asked 5,000 times over seven hours and got the same 409 every time.
|
||||
func TestRefusedCommitParksUntilSomethingChanges(t *testing.T) {
|
||||
var commits int
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case strings.HasSuffix(r.URL.Path, "/handoff"):
|
||||
commits++
|
||||
http.Error(w, "lease not owned", http.StatusConflict)
|
||||
case strings.HasSuffix(r.URL.Path, "/events"):
|
||||
_, _ = w.Write([]byte(`{"cursor":0,"events":[{"seq":1,"id":"c","type":"TaskCorrected","task_id":"t","version":9,"payload":{"state":"queued"},"surface":"web"}]}`))
|
||||
default:
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
}))
|
||||
defer s.Close()
|
||||
tx := releaseTransaction{ID: "tx", Phase: "anchor_pushed", Ref: "sha256:abc", AnchorSHA: "abc", LeaseEpoch: "e1"}
|
||||
w := &worker{
|
||||
api: federation.Client{BaseURL: s.URL, WorkerID: "h", Token: "t"},
|
||||
harnessID: "h",
|
||||
tasks: map[string]domain.Task{"t": {ID: "t"}},
|
||||
sessions: map[string]herdr.Session{"t": {PaneID: "pane"}},
|
||||
leases: map[string]lease{},
|
||||
releases: map[string]releaseTransaction{"t": tx},
|
||||
quarantined: map[string]bool{},
|
||||
statePath: t.TempDir() + "/state.json",
|
||||
hard: .75,
|
||||
}
|
||||
w.advanceRelease(context.Background(), "t", w.sessions["t"])
|
||||
w.advanceRelease(context.Background(), "t", w.sessions["t"])
|
||||
if commits != 1 {
|
||||
t.Fatalf("refused commit retried %d times without waiting", commits)
|
||||
}
|
||||
if w.releases["t"].NextAttemptAt.IsZero() {
|
||||
t.Fatal("refused commit was not parked")
|
||||
}
|
||||
// A reopen arrives as TaskCorrected. Any event about the task is the
|
||||
// change the parked commit was waiting for, so the same tick retries it
|
||||
// and, still refused, parks it again.
|
||||
if err := w.once(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if commits != 2 {
|
||||
t.Fatalf("an event about the task did not un-park its commit, commits=%d", commits)
|
||||
}
|
||||
if w.releases["t"].NextAttemptAt.IsZero() {
|
||||
t.Fatal("the second refusal did not park it again")
|
||||
}
|
||||
}
|
||||
|
||||
// The opposite case, and the one a backoff must not break: the coordinator is
|
||||
// unreachable or broken rather than answering. That says nothing about who
|
||||
// owns the task, so it has to retry at once.
|
||||
func TestTransientCommitFailureKeepsRetryingAtOnce(t *testing.T) {
|
||||
var commits int
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/handoff") {
|
||||
commits++
|
||||
http.Error(w, "upstream unavailable", http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}))
|
||||
defer s.Close()
|
||||
w := &worker{
|
||||
api: federation.Client{BaseURL: s.URL, WorkerID: "h", Token: "t"},
|
||||
harnessID: "h",
|
||||
tasks: map[string]domain.Task{"t": {ID: "t"}},
|
||||
sessions: map[string]herdr.Session{"t": {PaneID: "pane"}},
|
||||
leases: map[string]lease{},
|
||||
releases: map[string]releaseTransaction{"t": {ID: "tx", Phase: "anchor_pushed", Ref: "sha256:abc", AnchorSHA: "abc", LeaseEpoch: "e1"}},
|
||||
quarantined: map[string]bool{},
|
||||
statePath: t.TempDir() + "/state.json",
|
||||
hard: .75,
|
||||
}
|
||||
w.advanceRelease(context.Background(), "t", w.sessions["t"])
|
||||
w.advanceRelease(context.Background(), "t", w.sessions["t"])
|
||||
if commits != 2 {
|
||||
t.Fatalf("transport failure was parked like a refusal, commits=%d", commits)
|
||||
}
|
||||
if !w.releases["t"].NextAttemptAt.IsZero() {
|
||||
t.Fatal("transport failure must not park the transaction")
|
||||
}
|
||||
}
|
||||
|
||||
// Completion is terminal for a release transaction just as failure is. The
|
||||
// task is done; nothing will ever lease it again to pick the anchor up.
|
||||
func TestCompletedTaskDropsItsReleaseTransaction(t *testing.T) {
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/events") {
|
||||
_, _ = w.Write([]byte(`{"cursor":0,"events":[{"seq":1,"id":"c","type":"TaskCompleted","task_id":"t","version":9,"payload":{"report_ref":"sha256:r"},"surface":"system"}]}`))
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}))
|
||||
defer s.Close()
|
||||
w := &worker{
|
||||
api: federation.Client{BaseURL: s.URL, WorkerID: "h", Token: "t"},
|
||||
harnessID: "h",
|
||||
backend: deadTmuxBackend(t),
|
||||
tasks: map[string]domain.Task{"t": {ID: "t"}},
|
||||
sessions: map[string]herdr.Session{"t": {PaneID: "pane"}},
|
||||
leases: map[string]lease{},
|
||||
releases: map[string]releaseTransaction{"t": {ID: "tx", Phase: "anchor_pushed", Ref: "sha256:abc", AnchorSHA: "abc", LeaseEpoch: "e1"}},
|
||||
quarantined: map[string]bool{},
|
||||
statePath: t.TempDir() + "/state.json",
|
||||
hard: .75,
|
||||
}
|
||||
if err := w.once(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(w.releases) != 0 || len(w.sessions) != 0 {
|
||||
t.Fatalf("completed task kept its release: releases=%v sessions=%v", w.releases, w.sessions)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -445,6 +445,17 @@ func main() {
|
||||
}
|
||||
json.NewEncoder(w).Encode(s.Events(n))
|
||||
})
|
||||
mux.HandleFunc("/v1/debt", func(w http.ResponseWriter, r *http.Request) {
|
||||
// Read-only, and deliberately so. This projection is evidence about
|
||||
// history, not a new kind of truth: nothing here writes an event,
|
||||
// schedules work, or decides that something is worth repairing.
|
||||
ledger := s.DebtLedger()
|
||||
out := struct {
|
||||
domain.DebtLedger
|
||||
Eligible []operations.DebtCandidate `json:"eligible"`
|
||||
}{ledger, operations.EligibleDebt(ledger)}
|
||||
json.NewEncoder(w).Encode(out)
|
||||
})
|
||||
mux.HandleFunc("/v1/handoffs", func(w http.ResponseWriter, r *http.Request) {
|
||||
out := make([]domain.Event, 0)
|
||||
for _, e := range s.Events(0) {
|
||||
|
||||
@@ -0,0 +1,260 @@
|
||||
package domain
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"orchestra/internal/review"
|
||||
)
|
||||
|
||||
// Debt is the projected read model for accumulated cost: the defects,
|
||||
// workarounds and operational burdens that real tasks keep paying for. It is
|
||||
// derived from the event log rather than written by hand, so a claim in the
|
||||
// ledger can always be checked against the events that produced it.
|
||||
//
|
||||
// Four classes, and no more. A fifth invented at runtime makes the priority
|
||||
// model meaningless, because the thresholds in operations.CheckDebtEligibility
|
||||
// are stated per class.
|
||||
type DebtClass string
|
||||
|
||||
const (
|
||||
// DebtCorrectness is behaviour that is wrong or violates an invariant.
|
||||
DebtCorrectness DebtClass = "correctness"
|
||||
// DebtOperational works, but repeatedly costs time to diagnose, recover,
|
||||
// deploy, observe or operate.
|
||||
DebtOperational DebtClass = "operational"
|
||||
// DebtStructural is duplication or architecture demonstrably raising the
|
||||
// cost of future changes.
|
||||
DebtStructural DebtClass = "structural"
|
||||
// DebtPolish has no demonstrated cost yet, and never promotes itself.
|
||||
DebtPolish DebtClass = "polish"
|
||||
)
|
||||
|
||||
func (c DebtClass) Valid() bool {
|
||||
switch c {
|
||||
case DebtCorrectness, DebtOperational, DebtStructural, DebtPolish:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
type DebtStatus string
|
||||
|
||||
const (
|
||||
DebtObserved DebtStatus = "observed"
|
||||
DebtEligible DebtStatus = "eligible"
|
||||
DebtScheduled DebtStatus = "scheduled"
|
||||
DebtRepaired DebtStatus = "repaired"
|
||||
DebtWithdrawn DebtStatus = "withdrawn"
|
||||
)
|
||||
|
||||
// ObservationKind names where one piece of evidence came from. It is the kind
|
||||
// of the source fact, never an interpretation of it.
|
||||
type ObservationKind string
|
||||
|
||||
const (
|
||||
ObservationBlockReason ObservationKind = "block_reason"
|
||||
ObservationFailureClass ObservationKind = "failure_class"
|
||||
ObservationReviewFinding ObservationKind = "review_finding"
|
||||
ObservationPlanMismatch ObservationKind = "plan_mismatch"
|
||||
ObservationDeferredFinding ObservationKind = "deferred_finding"
|
||||
ObservationManualIntervention ObservationKind = "manual_intervention"
|
||||
ObservationWorkerFailure ObservationKind = "worker_observation"
|
||||
)
|
||||
|
||||
// SignatureVersion prefixes every signature this build produces. Normalization
|
||||
// rules will change, and without a version a change silently regroups every
|
||||
// historical observation, moving the recurrence counts that eligibility was
|
||||
// already decided on. A v2 signature never matches a v1 one, so old evidence
|
||||
// keeps the grouping it was counted under.
|
||||
const SignatureVersion = "v1"
|
||||
|
||||
// DebtSignature is the mechanical dedup key. Every part is a typed fact, never
|
||||
// prose, because exact signature match is the only path that attaches evidence
|
||||
// without a human. An empty part becomes "-" so the arity never varies.
|
||||
func DebtSignature(class DebtClass, reason, scope, component string) string {
|
||||
part := func(s string) string {
|
||||
s = strings.TrimSpace(strings.ToLower(s))
|
||||
s = strings.ReplaceAll(s, ":", "_")
|
||||
if s == "" {
|
||||
return "-"
|
||||
}
|
||||
return s
|
||||
}
|
||||
return strings.Join([]string{SignatureVersion, part(string(class)), part(reason), part(scope), part(component)}, ":")
|
||||
}
|
||||
|
||||
// DebtComponent reduces a repository path to the unit that owns it. Two
|
||||
// segments is the whole rule: it keeps internal/store distinct from
|
||||
// internal/herdr without splitting one package across files.
|
||||
func DebtComponent(path string) string {
|
||||
path = strings.TrimSpace(strings.Trim(path, "/"))
|
||||
if path == "" {
|
||||
return ""
|
||||
}
|
||||
parts := strings.Split(path, "/")
|
||||
if len(parts) == 1 {
|
||||
return parts[0]
|
||||
}
|
||||
return parts[0] + "/" + parts[1]
|
||||
}
|
||||
|
||||
// DebtObservation is one piece of counted evidence. Provenance is required and
|
||||
// is either an event id or, for history imported from a markdown ledger that
|
||||
// predates this projection, an explicit legacy reference. Exactly one, because
|
||||
// a fabricated event id would break the rule the ledger exists to enforce.
|
||||
type DebtObservation struct {
|
||||
EventID string `json:"event_id,omitempty"`
|
||||
LegacyRef string `json:"legacy_ref,omitempty"`
|
||||
TaskID string `json:"task_id,omitempty"`
|
||||
Kind ObservationKind `json:"kind"`
|
||||
Signature string `json:"signature"`
|
||||
Detail string `json:"detail,omitempty"`
|
||||
Paths []string `json:"paths,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
func (o DebtObservation) Validate() error {
|
||||
if (o.EventID == "") == (o.LegacyRef == "") {
|
||||
return fmt.Errorf("observation needs exactly one of event_id and legacy_ref")
|
||||
}
|
||||
if o.Kind == "" {
|
||||
return fmt.Errorf("observation kind required")
|
||||
}
|
||||
if !strings.HasPrefix(o.Signature, SignatureVersion+":") {
|
||||
return fmt.Errorf("observation signature %q is not %s", o.Signature, SignatureVersion)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DebtItem groups observations that share a signature. Every count the
|
||||
// priority model needs is derived from Observations rather than stored, so a
|
||||
// count can never drift from the log that justifies it.
|
||||
type DebtItem struct {
|
||||
ID string `json:"id"`
|
||||
Class DebtClass `json:"class"`
|
||||
Status DebtStatus `json:"status"`
|
||||
Symptom string `json:"symptom"`
|
||||
Consequence string `json:"consequence,omitempty"`
|
||||
Severity review.Severity `json:"severity,omitempty"`
|
||||
Signatures []string `json:"signatures"`
|
||||
Paths []string `json:"paths,omitempty"`
|
||||
Observations []DebtObservation `json:"observations"`
|
||||
IntroducedIn string `json:"introduced_in,omitempty"`
|
||||
RepairBoundary string `json:"repair_boundary,omitempty"`
|
||||
RepairTask string `json:"repair_task,omitempty"`
|
||||
RepairCommit string `json:"repair_commit,omitempty"`
|
||||
}
|
||||
|
||||
// Recurrence is how many times this shape has been seen.
|
||||
func (d DebtItem) Recurrence() int { return len(d.Observations) }
|
||||
|
||||
// AffectedTasks is how broadly the shape has spread. Distinct tasks, because
|
||||
// one task failing ten times is weaker evidence than ten tasks failing once.
|
||||
func (d DebtItem) AffectedTasks() int {
|
||||
seen := map[string]bool{}
|
||||
for _, o := range d.Observations {
|
||||
if o.TaskID != "" {
|
||||
seen[o.TaskID] = true
|
||||
}
|
||||
}
|
||||
return len(seen)
|
||||
}
|
||||
|
||||
// BlockedTasks counts distinct tasks this shape actually stopped.
|
||||
func (d DebtItem) BlockedTasks() int {
|
||||
seen := map[string]bool{}
|
||||
for _, o := range d.Observations {
|
||||
if o.Kind == ObservationBlockReason && o.TaskID != "" {
|
||||
seen[o.TaskID] = true
|
||||
}
|
||||
}
|
||||
return len(seen)
|
||||
}
|
||||
|
||||
// ManualInterventions counts recorded operator repairs. It reads zero on any
|
||||
// log written before that recording exists, which is why the ledger reports it
|
||||
// as an evidence gap rather than as an absence of operator cost.
|
||||
func (d DebtItem) ManualInterventions() int {
|
||||
n := 0
|
||||
for _, o := range d.Observations {
|
||||
if o.Kind == ObservationManualIntervention {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// Components is the breadth of the shape across the tree.
|
||||
func (d DebtItem) Components() []string {
|
||||
seen := map[string]bool{}
|
||||
out := []string{}
|
||||
for _, o := range d.Observations {
|
||||
for _, p := range o.Paths {
|
||||
if c := DebtComponent(p); c != "" && !seen[c] {
|
||||
seen[c] = true
|
||||
out = append(out, c)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Imported reports whether any evidence here came from a markdown ledger
|
||||
// rather than from an event. A reader can then tell a counted fact from an
|
||||
// imported claim without opening the observations.
|
||||
func (d DebtItem) Imported() bool {
|
||||
for _, o := range d.Observations {
|
||||
if o.LegacyRef != "" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// EvidenceGap is a debt signal the ledger knows it cannot see. Reporting the
|
||||
// gap is the point: a ledger that silently omits what it cannot record reads
|
||||
// as "no operator cost" when it means "operator cost is not recorded".
|
||||
type EvidenceGap struct {
|
||||
Kind ObservationKind `json:"kind"`
|
||||
Reason string `json:"reason"`
|
||||
// Durable is false when no event type carries this evidence at all. It is
|
||||
// true when the log could carry it and this particular log does not.
|
||||
Durable bool `json:"durable"`
|
||||
}
|
||||
|
||||
// DebtLedger is the whole read model: what history establishes, and what it
|
||||
// cannot.
|
||||
type DebtLedger struct {
|
||||
Items []DebtItem `json:"items"`
|
||||
Gaps []EvidenceGap `json:"gaps"`
|
||||
// Events is how many log entries the projection folded, so a caller can
|
||||
// tell an empty ledger from an unread log.
|
||||
Events int `json:"events"`
|
||||
}
|
||||
|
||||
// DebtClassForBlockReason maps a typed block reason to a class. A reason that
|
||||
// is a normal lifecycle stop rather than a cost returns false: waiting for a
|
||||
// human decision is the system working, not debt.
|
||||
func DebtClassForBlockReason(r BlockReason) (DebtClass, bool) {
|
||||
switch r {
|
||||
case BlockReasonLeaseFailure, BlockReasonWorkerOffline, BlockReasonLeaseExpired, BlockReasonSystem:
|
||||
return DebtOperational, true
|
||||
case BlockReasonHandoffValidation, BlockReasonPlanMismatch:
|
||||
return DebtCorrectness, true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// DebtClassForFailureClass maps a worker failure class to a class. Only the
|
||||
// classes a worker actually emits are listed; an unknown one is not guessed at.
|
||||
func DebtClassForFailureClass(f string) (DebtClass, bool) {
|
||||
switch f {
|
||||
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired":
|
||||
return DebtOperational, true
|
||||
case "invalid_handoff":
|
||||
return DebtCorrectness, true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package domain
|
||||
|
||||
import "testing"
|
||||
|
||||
// Normalization rules will change. A v2 signature must never match a v1 one,
|
||||
// or changing them silently regroups history and moves the recurrence counts
|
||||
// eligibility was already decided on.
|
||||
func TestSignatureCarriesItsVersion(t *testing.T) {
|
||||
sig := DebtSignature(DebtOperational, "lease_expired", "workpc-opencode", "internal/herdr")
|
||||
if sig != "v1:operational:lease_expired:workpc-opencode:internal/herdr" {
|
||||
t.Fatalf("signature %q", sig)
|
||||
}
|
||||
// Arity never varies, so a missing part cannot shift the fields left.
|
||||
if got := DebtSignature(DebtPolish, "deferred_finding", "", ""); got != "v1:polish:deferred_finding:-:-" {
|
||||
t.Fatalf("empty parts not padded: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComponentIsTwoSegments(t *testing.T) {
|
||||
for path, want := range map[string]string{
|
||||
"internal/store/store.go": "internal/store",
|
||||
"internal/herdr/adapter.go": "internal/herdr",
|
||||
"cmd/orchestra-worker/main.go": "cmd/orchestra-worker",
|
||||
"BURNIN.md": "BURNIN.md",
|
||||
"": "",
|
||||
} {
|
||||
if got := DebtComponent(path); got != want {
|
||||
t.Fatalf("component(%q) = %q, want %q", path, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Imported history predates the events that would have justified it. A
|
||||
// fabricated event id would break the provenance rule the ledger enforces.
|
||||
func TestObservationNeedsExactlyOneProvenance(t *testing.T) {
|
||||
sig := DebtSignature(DebtOperational, "x", "", "")
|
||||
both := DebtObservation{EventID: "e1", LegacyRef: "BURNIN.md:F18", Kind: ObservationBlockReason, Signature: sig}
|
||||
neither := DebtObservation{Kind: ObservationBlockReason, Signature: sig}
|
||||
if both.Validate() == nil || neither.Validate() == nil {
|
||||
t.Fatal("exactly one of event_id and legacy_ref must be required")
|
||||
}
|
||||
for _, ok := range []DebtObservation{
|
||||
{EventID: "e1", Kind: ObservationBlockReason, Signature: sig},
|
||||
{LegacyRef: "BURNIN.md:F18", Kind: ObservationBlockReason, Signature: sig},
|
||||
} {
|
||||
if err := ok.Validate(); err != nil {
|
||||
t.Fatalf("valid observation refused: %v", err)
|
||||
}
|
||||
}
|
||||
stale := DebtObservation{EventID: "e1", Kind: ObservationBlockReason, Signature: "v0:operational:x:-:-"}
|
||||
if stale.Validate() == nil {
|
||||
t.Fatal("a signature from another version must be refused")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package operations
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"orchestra/internal/domain"
|
||||
)
|
||||
|
||||
// DebtCheck is why a debt item may or may not become work. Reasons are listed
|
||||
// rather than summarised, for the same reason SubmissionCheck lists them: "not
|
||||
// eligible" alone sends an operator reading code.
|
||||
type DebtCheck struct {
|
||||
Eligible bool `json:"eligible"`
|
||||
Reasons []string `json:"reasons,omitempty"`
|
||||
}
|
||||
|
||||
// CheckDebtEligibility is the whole promotion rule, as one pure function of a
|
||||
// projected item. It decides only whether a maintenance task may be created.
|
||||
// It never creates one, and it never changes what that task must then pass:
|
||||
// classification as debt changes what gets scheduled, never what gets checked.
|
||||
//
|
||||
// The thresholds differ by class on purpose. Correctness and operational debt
|
||||
// have already cost something measurable. Structural debt needs evidence that
|
||||
// it is causing repeated work rather than merely offending taste. Polish never
|
||||
// promotes itself, or the ledger becomes a permanent cleanup generator.
|
||||
func CheckDebtEligibility(item domain.DebtItem) DebtCheck {
|
||||
recurrence, tasks := item.Recurrence(), item.AffectedTasks()
|
||||
blocked, manual := item.BlockedTasks(), item.ManualInterventions()
|
||||
switch item.Class {
|
||||
case domain.DebtCorrectness:
|
||||
if recurrence >= 1 {
|
||||
return DebtCheck{true, []string{fmt.Sprintf("correctness debt is eligible on first confirmed observation, and has %d", recurrence)}}
|
||||
}
|
||||
return DebtCheck{false, []string{"no confirmed observation"}}
|
||||
case domain.DebtOperational:
|
||||
var why []string
|
||||
if manual >= 1 {
|
||||
why = append(why, fmt.Sprintf("%d manual intervention(s) recorded", manual))
|
||||
}
|
||||
if recurrence >= 3 && tasks >= 2 {
|
||||
why = append(why, fmt.Sprintf("recurred %d times across %d tasks", recurrence, tasks))
|
||||
}
|
||||
if len(why) > 0 {
|
||||
return DebtCheck{true, why}
|
||||
}
|
||||
return DebtCheck{false, []string{
|
||||
fmt.Sprintf("needs 3 occurrences across 2 tasks, or 1 manual intervention; has %d across %d tasks with %d interventions", recurrence, tasks, manual),
|
||||
"manual interventions are not recorded by any event type, so that count reads 0 on every current log",
|
||||
}}
|
||||
case domain.DebtStructural:
|
||||
if recurrence >= 3 {
|
||||
return DebtCheck{true, []string{fmt.Sprintf("%d review findings in this component", recurrence)}}
|
||||
}
|
||||
if blocked >= 2 {
|
||||
return DebtCheck{true, []string{fmt.Sprintf("blocked %d distinct tasks in this component", blocked)}}
|
||||
}
|
||||
return DebtCheck{false, []string{
|
||||
fmt.Sprintf("needs 3 review findings or 2 blocked tasks in one component; has %d findings and %d blocked", recurrence, blocked),
|
||||
}}
|
||||
case domain.DebtPolish:
|
||||
return DebtCheck{false, []string{"polish never promotes itself, an operator promotes it explicitly"}}
|
||||
}
|
||||
return DebtCheck{false, []string{"unknown debt class " + string(item.Class)}}
|
||||
}
|
||||
|
||||
// EligibleDebt filters a projected ledger to what policy would allow to become
|
||||
// work. It returns the check alongside each item so the reasons stay visible.
|
||||
func EligibleDebt(ledger domain.DebtLedger) []DebtCandidate {
|
||||
out := []DebtCandidate{}
|
||||
for _, item := range ledger.Items {
|
||||
if check := CheckDebtEligibility(item); check.Eligible {
|
||||
out = append(out, DebtCandidate{Item: item, Check: check})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type DebtCandidate struct {
|
||||
Item domain.DebtItem `json:"item"`
|
||||
Check DebtCheck `json:"check"`
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package operations
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"orchestra/internal/domain"
|
||||
)
|
||||
|
||||
func debtItem(class domain.DebtClass, kinds ...struct {
|
||||
kind domain.ObservationKind
|
||||
task string
|
||||
}) domain.DebtItem {
|
||||
item := domain.DebtItem{Class: class}
|
||||
for _, k := range kinds {
|
||||
item.Observations = append(item.Observations, domain.DebtObservation{Kind: k.kind, TaskID: k.task})
|
||||
}
|
||||
return item
|
||||
}
|
||||
|
||||
type obs = struct {
|
||||
kind domain.ObservationKind
|
||||
task string
|
||||
}
|
||||
|
||||
// The thresholds are the whole policy, so each class gets its own case. Polish
|
||||
// is the one that must never pass, or the ledger becomes a cleanup generator.
|
||||
func TestDebtEligibilityPerClass(t *testing.T) {
|
||||
fail := obs{domain.ObservationFailureClass, "t1"}
|
||||
if !CheckDebtEligibility(debtItem(domain.DebtCorrectness, fail)).Eligible {
|
||||
t.Fatal("correctness debt is eligible on first observation")
|
||||
}
|
||||
twice := debtItem(domain.DebtOperational, fail, obs{domain.ObservationFailureClass, "t1"})
|
||||
if CheckDebtEligibility(twice).Eligible {
|
||||
t.Fatal("two occurrences on one task must not qualify as operational debt")
|
||||
}
|
||||
spread := debtItem(domain.DebtOperational, fail,
|
||||
obs{domain.ObservationFailureClass, "t2"}, obs{domain.ObservationFailureClass, "t3"})
|
||||
if !CheckDebtEligibility(spread).Eligible {
|
||||
t.Fatal("three occurrences across three tasks must qualify")
|
||||
}
|
||||
manual := debtItem(domain.DebtOperational, obs{domain.ObservationManualIntervention, "t1"})
|
||||
if !CheckDebtEligibility(manual).Eligible {
|
||||
t.Fatal("one manual intervention must qualify on its own")
|
||||
}
|
||||
blocked := debtItem(domain.DebtStructural,
|
||||
obs{domain.ObservationBlockReason, "t1"}, obs{domain.ObservationBlockReason, "t2"})
|
||||
if !CheckDebtEligibility(blocked).Eligible {
|
||||
t.Fatal("two blocked tasks in one component must qualify as structural")
|
||||
}
|
||||
polish := debtItem(domain.DebtPolish, fail, fail, fail, fail, fail)
|
||||
if CheckDebtEligibility(polish).Eligible {
|
||||
t.Fatal("polish must never promote itself, at any recurrence")
|
||||
}
|
||||
}
|
||||
|
||||
// A refusal has to say what is missing. "Not eligible" alone sends an operator
|
||||
// reading code, which is the mistake SubmissionCheck already documents.
|
||||
func TestDebtRefusalNamesTheMissingEvidence(t *testing.T) {
|
||||
check := CheckDebtEligibility(debtItem(domain.DebtOperational, obs{domain.ObservationFailureClass, "t1"}))
|
||||
if check.Eligible || len(check.Reasons) == 0 {
|
||||
t.Fatalf("want a refusal with reasons, got %+v", check)
|
||||
}
|
||||
joined := strings.Join(check.Reasons, " ")
|
||||
if !strings.Contains(joined, "manual interventions are not recorded") {
|
||||
t.Fatalf("the refusal must say the intervention count is structurally zero: %v", check.Reasons)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"orchestra/internal/domain"
|
||||
"orchestra/internal/review"
|
||||
)
|
||||
|
||||
// ProjectDebt folds canonical history into a debt ledger. It is a pure
|
||||
// function of the events so it can be tested against a real log, and it writes
|
||||
// nothing: this projection is a read model, not a new kind of truth.
|
||||
//
|
||||
// It deliberately reports what it cannot see. Worker observations and operator
|
||||
// interventions are not carried by any event type today, so a ledger that
|
||||
// stayed silent about them would read as "no operational cost" when it means
|
||||
// "operational cost is unrecorded".
|
||||
func ProjectDebt(events []domain.Event) domain.DebtLedger {
|
||||
return projectDebt(events, nil)
|
||||
}
|
||||
|
||||
// ProjectDebtWithArtifacts is ProjectDebt plus the CAS reads that review
|
||||
// findings need. Findings are sealed as an artifact rather than inlined in the
|
||||
// event, and they are the only evidence structural debt has, so a projection
|
||||
// without them can never produce a structural item at all.
|
||||
func ProjectDebtWithArtifacts(events []domain.Event, read func(ref string) ([]byte, error)) domain.DebtLedger {
|
||||
return projectDebt(events, read)
|
||||
}
|
||||
|
||||
func projectDebt(events []domain.Event, readArtifact func(string) ([]byte, error)) domain.DebtLedger {
|
||||
bySignature := map[string]*domain.DebtItem{}
|
||||
order := []string{}
|
||||
seenKinds := map[domain.ObservationKind]bool{}
|
||||
|
||||
add := func(class domain.DebtClass, sig string, o domain.DebtObservation, symptom string, sev review.Severity) {
|
||||
o.Signature = sig
|
||||
if o.Validate() != nil {
|
||||
return
|
||||
}
|
||||
seenKinds[o.Kind] = true
|
||||
item, ok := bySignature[sig]
|
||||
if !ok {
|
||||
item = &domain.DebtItem{
|
||||
// The id is the signature, not an Fxx number. This projection
|
||||
// produces candidates from history; adopting a curated id is a
|
||||
// later, deliberate step, and minting one here would collide
|
||||
// with the hand-written ledger.
|
||||
ID: sig, Class: class, Status: domain.DebtObserved,
|
||||
Symptom: symptom, Signatures: []string{sig}, Severity: sev,
|
||||
}
|
||||
bySignature[sig] = item
|
||||
order = append(order, sig)
|
||||
}
|
||||
item.Observations = append(item.Observations, o)
|
||||
for _, p := range o.Paths {
|
||||
if !contains(item.Paths, p) {
|
||||
item.Paths = append(item.Paths, p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for _, e := range events {
|
||||
var p map[string]any
|
||||
if len(e.Payload) > 0 && json.Unmarshal(e.Payload, &p) != nil {
|
||||
continue
|
||||
}
|
||||
base := domain.DebtObservation{EventID: e.ID, TaskID: e.TaskID, At: e.At}
|
||||
harness, _ := p["harness_id"].(string)
|
||||
switch e.Type {
|
||||
case "TaskBlocked":
|
||||
reason, _ := p["block_reason"].(string)
|
||||
if reason == "" {
|
||||
reason = string(domain.InferBlockReason(str(p["blocker"])))
|
||||
}
|
||||
class, ok := domain.DebtClassForBlockReason(domain.BlockReason(reason))
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
o := base
|
||||
o.Kind = domain.ObservationBlockReason
|
||||
o.Detail = str(p["blocker"])
|
||||
add(class, domain.DebtSignature(class, reason, harness, "lease"), o,
|
||||
"tasks are blocked with "+reason, review.Important)
|
||||
case "TaskFailed", "TaskReleased":
|
||||
// A release carries a failure class only when it is a reclaim. A
|
||||
// handoff release is ordinary progress and never debt.
|
||||
failure, _ := p["failure_class"].(string)
|
||||
if failure == "" {
|
||||
failure, _ = p["reason"].(string)
|
||||
}
|
||||
if e.Type == "TaskReleased" && str(p["handoff_ref"]) != "" {
|
||||
continue
|
||||
}
|
||||
class, ok := domain.DebtClassForFailureClass(failure)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
o := base
|
||||
o.Kind = domain.ObservationFailureClass
|
||||
o.Detail = str(p["last_error"])
|
||||
add(class, domain.DebtSignature(class, failure, harness, "lease"), o,
|
||||
"tasks end in "+failure, review.Important)
|
||||
case domain.EventPlanMismatchRecorded:
|
||||
o := base
|
||||
o.Kind = domain.ObservationPlanMismatch
|
||||
o.Detail = str(p["observed"])
|
||||
o.Paths = strList(p["evidence"])
|
||||
component := firstComponent(o.Paths)
|
||||
add(domain.DebtCorrectness,
|
||||
domain.DebtSignature(domain.DebtCorrectness, "plan_mismatch", "", component), o,
|
||||
"the repository contradicts sealed plans", review.Important)
|
||||
case domain.EventReviewRecorded:
|
||||
// Repeated findings in one component are the only mechanical
|
||||
// evidence that structure is costing future changes. One finding
|
||||
// is a review doing its job.
|
||||
if readArtifact == nil {
|
||||
break
|
||||
}
|
||||
raw, err := readArtifact(str(p["artifact_ref"]))
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
result, err := review.Decode(raw)
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
for _, f := range result.Findings {
|
||||
component := domain.DebtComponent(f.File)
|
||||
o := base
|
||||
o.Kind = domain.ObservationReviewFinding
|
||||
o.Detail = f.Claim
|
||||
if f.File != "" {
|
||||
o.Paths = []string{f.File}
|
||||
}
|
||||
add(domain.DebtStructural,
|
||||
domain.DebtSignature(domain.DebtStructural, string(f.Severity), "", component), o,
|
||||
"review keeps finding "+string(f.Severity)+" issues in "+component, f.Severity)
|
||||
}
|
||||
case domain.EventDeferredFindingRecorded:
|
||||
o := base
|
||||
o.Kind = domain.ObservationDeferredFinding
|
||||
o.Detail = str(p["summary"])
|
||||
// Polish until something demonstrates a cost. A discovery nobody
|
||||
// has paid for yet is not debt with a priority.
|
||||
add(domain.DebtPolish,
|
||||
domain.DebtSignature(domain.DebtPolish, "deferred_finding", "", ""), o,
|
||||
"out-of-scope discoveries recorded and unaddressed", review.Minor)
|
||||
}
|
||||
}
|
||||
|
||||
items := make([]domain.DebtItem, 0, len(order))
|
||||
for _, sig := range order {
|
||||
items = append(items, *bySignature[sig])
|
||||
}
|
||||
sort.SliceStable(items, func(i, j int) bool {
|
||||
if items[i].Recurrence() != items[j].Recurrence() {
|
||||
return items[i].Recurrence() > items[j].Recurrence()
|
||||
}
|
||||
return items[i].ID < items[j].ID
|
||||
})
|
||||
return domain.DebtLedger{Items: items, Gaps: debtGaps(seenKinds), Events: len(events)}
|
||||
}
|
||||
|
||||
// debtGaps separates two different silences. A kind no event type carries is a
|
||||
// hole in the system. A kind the log could carry and does not is a fact about
|
||||
// this history.
|
||||
func debtGaps(seen map[domain.ObservationKind]bool) []domain.EvidenceGap {
|
||||
gaps := []domain.EvidenceGap{
|
||||
{Kind: domain.ObservationManualIntervention, Durable: false,
|
||||
Reason: "no event type records an operator repair, so every manual recovery is invisible to this ledger"},
|
||||
{Kind: domain.ObservationWorkerFailure, Durable: false,
|
||||
Reason: "worker observations live in worker memory and reach the coordinator only inside WorkerHealth, which is not persisted"},
|
||||
}
|
||||
for _, k := range []domain.ObservationKind{
|
||||
domain.ObservationBlockReason, domain.ObservationFailureClass,
|
||||
domain.ObservationReviewFinding, domain.ObservationPlanMismatch,
|
||||
domain.ObservationDeferredFinding,
|
||||
} {
|
||||
if !seen[k] {
|
||||
gaps = append(gaps, domain.EvidenceGap{Kind: k, Durable: true,
|
||||
Reason: "carried by the log, but this history contains none"})
|
||||
}
|
||||
}
|
||||
return gaps
|
||||
}
|
||||
|
||||
// DebtLedger projects the whole log. Read-only, like every other projection
|
||||
// the coordinator serves.
|
||||
func (s *Store) DebtLedger() domain.DebtLedger {
|
||||
return ProjectDebtWithArtifacts(s.Events(0), s.Artifact)
|
||||
}
|
||||
|
||||
func str(v any) string { s, _ := v.(string); return s }
|
||||
|
||||
func strList(v any) []string {
|
||||
list, ok := v.([]any)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(list))
|
||||
for _, item := range list {
|
||||
if s := strings.TrimSpace(str(item)); s != "" {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func firstComponent(paths []string) string {
|
||||
for _, p := range paths {
|
||||
if c := domain.DebtComponent(p); c != "" {
|
||||
return c
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func contains(list []string, s string) bool {
|
||||
for _, item := range list {
|
||||
if item == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"orchestra/internal/domain"
|
||||
)
|
||||
|
||||
func debtEvent(id, typ, task string, payload map[string]any) domain.Event {
|
||||
b, _ := json.Marshal(payload)
|
||||
return domain.Event{ID: id, Type: typ, TaskID: task, Payload: b, At: time.Now().UTC()}
|
||||
}
|
||||
|
||||
// The opencode shape from run 14: one harness failing the same way across
|
||||
// several tasks. That is the recurrence the ledger exists to notice.
|
||||
func TestProjectDebtGroupsOneFailureShapeAcrossTasks(t *testing.T) {
|
||||
events := []domain.Event{
|
||||
debtEvent("e1", "TaskFailed", "t1", map[string]any{"failure_class": "retry_limit", "harness_id": "workpc-opencode"}),
|
||||
debtEvent("e2", "TaskFailed", "t2", map[string]any{"failure_class": "retry_limit", "harness_id": "workpc-opencode"}),
|
||||
debtEvent("e3", "TaskFailed", "t3", map[string]any{"failure_class": "retry_limit", "harness_id": "workpc-opencode"}),
|
||||
debtEvent("e4", "TaskFailed", "t4", map[string]any{"failure_class": "retry_limit", "harness_id": "workpc-claude"}),
|
||||
// A handoff release is ordinary progress and must not become debt.
|
||||
debtEvent("e5", "TaskReleased", "t5", map[string]any{"handoff_ref": "sha256:abc", "harness_id": "workpc-claude"}),
|
||||
}
|
||||
ledger := ProjectDebt(events)
|
||||
if len(ledger.Items) != 2 {
|
||||
t.Fatalf("want one item per harness, got %d: %+v", len(ledger.Items), ledger.Items)
|
||||
}
|
||||
top := ledger.Items[0]
|
||||
if top.Recurrence() != 3 || top.AffectedTasks() != 3 {
|
||||
t.Fatalf("recurrence %d across %d tasks, want 3 and 3", top.Recurrence(), top.AffectedTasks())
|
||||
}
|
||||
if top.Class != domain.DebtOperational {
|
||||
t.Fatalf("class %q, want operational", top.Class)
|
||||
}
|
||||
if got := top.ID; got != "v1:operational:retry_limit:workpc-opencode:lease" {
|
||||
t.Fatalf("signature %q", got)
|
||||
}
|
||||
for _, o := range top.Observations {
|
||||
if o.EventID == "" || o.LegacyRef != "" {
|
||||
t.Fatalf("observation lost its event provenance: %+v", o)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A lifecycle stop is the system working. Waiting for a human is not debt, and
|
||||
// counting it would drown the real signal.
|
||||
func TestProjectDebtIgnoresOrdinaryLifecycleStops(t *testing.T) {
|
||||
events := []domain.Event{
|
||||
debtEvent("e1", "TaskBlocked", "t1", map[string]any{"block_reason": "human_decision", "blocker": "which route"}),
|
||||
debtEvent("e2", "TaskBlocked", "t2", map[string]any{"block_reason": "trajectory_gate", "blocker": "confirm plan"}),
|
||||
debtEvent("e3", "TaskBlocked", "t3", map[string]any{"block_reason": "lease_expired", "blocker": "lease expired"}),
|
||||
}
|
||||
ledger := ProjectDebt(events)
|
||||
if len(ledger.Items) != 1 || ledger.Items[0].Class != domain.DebtOperational {
|
||||
t.Fatalf("want only the lease_expired item, got %+v", ledger.Items)
|
||||
}
|
||||
}
|
||||
|
||||
// Incompleteness is part of the result. A ledger that stays silent about what
|
||||
// it cannot record reads as "no operator cost" when it means "operator cost is
|
||||
// not recorded anywhere".
|
||||
func TestProjectDebtReportsWhatItCannotSee(t *testing.T) {
|
||||
ledger := ProjectDebt(nil)
|
||||
var manual, worker bool
|
||||
for _, g := range ledger.Gaps {
|
||||
if g.Durable {
|
||||
continue
|
||||
}
|
||||
switch g.Kind {
|
||||
case domain.ObservationManualIntervention:
|
||||
manual = true
|
||||
case domain.ObservationWorkerFailure:
|
||||
worker = true
|
||||
}
|
||||
}
|
||||
if !manual || !worker {
|
||||
t.Fatalf("the two known holes must always be reported: %+v", ledger.Gaps)
|
||||
}
|
||||
for _, g := range ledger.Gaps {
|
||||
if g.Reason == "" {
|
||||
t.Fatalf("gap %q has no reason", g.Kind)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -424,6 +424,13 @@ func (s *Store) apply(e domain.Event) error {
|
||||
t.Lease = nil
|
||||
case "TaskBlocked", "TaskNeedsAttention":
|
||||
if e.Type == "TaskBlocked" {
|
||||
if t.Lease != nil {
|
||||
// Same reason as TaskReleased: a worker may hold a pushed
|
||||
// anchor whose commit was refused. A reopen returns the task
|
||||
// to the queue, and the late-handoff path can only accept it
|
||||
// if the epoch that ended is still on record.
|
||||
t.LastLeaseEpoch = t.Lease.Epoch
|
||||
}
|
||||
t.State = domain.StateBlocked
|
||||
t.Lease = nil
|
||||
} else {
|
||||
|
||||
@@ -693,3 +693,33 @@ func TestExpiryRetainsLeaseEpoch(t *testing.T) {
|
||||
t.Fatalf("last lease epoch %q, want %q", after.LastLeaseEpoch, epoch)
|
||||
}
|
||||
}
|
||||
|
||||
// A worker can hold a pushed anchor whose commit was refused when an operator
|
||||
// blocks the task. A reopen returns it to the queue, and the late-handoff path
|
||||
// can only accept that exact owner if the epoch that ended is still recorded.
|
||||
func TestBlockRetainsLeaseEpochForALaterReopen(t *testing.T) {
|
||||
s, err := Open(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Append(created("e1")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
id := s.Tasks()[0].ID
|
||||
if _, err := s.Lease(id, "h1", time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
leased, _ := s.Task(id)
|
||||
epoch := leased.Lease.Epoch
|
||||
p, _ := json.Marshal(map[string]any{"blocker": "parked by the operator", "harness_id": "h1", "lease_epoch": epoch})
|
||||
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskBlocked", TaskID: id, Version: leased.Version + 1, Payload: p, Surface: string(authz.System)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
after, _ := s.Task(id)
|
||||
if after.State != domain.StateBlocked || after.Lease != nil {
|
||||
t.Fatalf("expected a blocked unleased task, got %s lease=%v", after.State, after.Lease)
|
||||
}
|
||||
if after.LastLeaseEpoch != epoch || epoch == "" {
|
||||
t.Fatalf("last lease epoch %q, want %q", after.LastLeaseEpoch, epoch)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user