7 Commits

Author SHA1 Message Date
kami 2d28f7b462 Record slice one's first run against 881 real events
Seven eligible items, three gaps, and four defects in the model that a
read-only projection surfaced before any schema was committed to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 02:02:34 +04:00
kami a757cffc78 Project a debt ledger from canonical history, read-only
Slice one of DEBT-DESIGN.md, with the four amendments applied. It writes
nothing: no new event types, no scheduling, no clustering, no
maintenance tasks. The point is to find out whether the model can
represent debt this project already knows about, before committing to a
durable schema.

The projected type lives in domain and the fold lives in store, so the
first implementation does not bake a read model into the command layer.
operations owns the one action that exists, CheckDebtEligibility, which
is a pure function returning explicit reasons like CheckSubmission.

Signatures carry their version in the string. Normalization rules will
change, and without a version that silently regroups history and moves
the recurrence counts eligibility was already decided on.

Observations require exactly one of event_id and legacy_ref. Imported
Fxx history predates the events that would justify it, and a fabricated
event id would break the provenance rule the ledger exists to enforce.

Incompleteness is reported, not hidden. Manual interventions and worker
observations are carried by no event type, so the ledger names both as
non-durable gaps rather than reading as "no operational cost". The
operational refusal reason says the intervention count is structurally
zero on every current log.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 02:00:58 +04:00
kami 2f61a99986 Design the debt ledger from what the event log already carries
Nine questions answered against the tree at 3c7cf95, no implementation.
The ledger is two event types, one projection beside Task, one pure
eligibility function and one read-only endpoint. Nothing in the task
lifecycle changes.

The first slice writes nothing: a projection over the existing log that
must reproduce the release loop, the opencode adapter gap and the
retry-idleness dynamic from runs 10 to 14. A model that cannot represent
debt already known is wrong before any schema is committed to.

Also corrects run 14: the OpenCode Zen free tier is not blocked, the
selected model is.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 01:51:53 +04:00
kami 595a1d3533 Record run 14: plan-phase execution proven, opencode diagnosed
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-29 01:11:57 +04:00
kami 0797432d6f Record run 13: F60 settled, and F18 populated under a real failure
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-28 23:55:20 +04:00
kami 3c7cf95d8c Settle a release transaction deterministically in every case
F60, and the general rule F58 and F59 were reaching for one case at a
time: a transaction must settle or be abandoned deterministically, and
must never spin on an answer that cannot change.

Terminal now means failed or completed. Both drop the transaction and
free the session; nothing will ever lease either task again.

Blocked keeps the transaction, because a reopen returns the task to the
queue and that exact owner can still commit. TaskBlocked therefore
retains the ending epoch the way TaskReleased already did, or the
late-handoff path would have nothing to fence against after the reopen.

A refusal parks the commit instead of retrying every five seconds. It
is the coordinator's answer about who owns the task, so it stays true
until an event about that task arrives, and any such event un-parks it.
A reopen arrives as TaskCorrected, so the rule cannot be a list of
event types. Backoff runs 30s to a 5 minute cap.

A transport failure is not an answer and keeps retrying at once. That
distinction is the whole reason the park keys on a 4xx StatusError
rather than on any error at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-28 23:52:18 +04:00
kami 0f83559ecc Record F18, the merge, and what run 12 leaves open
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVbaKucEYBjMqVeUgJUsc1
2026-08-28 23:45:03 +04:00
13 changed files with 1648 additions and 6 deletions
+253
View File
@@ -2274,3 +2274,256 @@ tmux -L orchestra kill-session -t <session>
Task `06G4JX6MSQEP7N0D5JWW9EP5X4` is the other run 12 task and is `in_review` on
a pull request. It is real work and should be reviewed or failed, not cleaned.
### F18 closed: a bounded observation ring
`d6ee10f`. `WorkerHealth` now carries up to sixteen distinct observations, each
with a repeat count and first/last times. Collapsing is by message, not by
position, because a loop interleaved with other failures would otherwise still
flush the ring. Eviction drops the least recently seen, so a loop keeps its
slot but carries the count that says what it is. `last_error` and `error_at`
keep their wire names and still report only the newest failure.
The ring lives in memory beside `last_error` and is not persisted, which is
what `last_error` already did across a restart.
Deployed and verified on both halves. It reports nothing yet, because
`omitempty` hides an empty ring and nothing has failed since the deploy. No UI
renders it: `internal/ui` never showed `last_error` either, so
`GET /v1/federation/workers` is the only reader today.
### Run 12 closed out
PR 17 merged, task `06G4JX6MSQEP7N0D5JWW9EP5X4` completed at v32, pane closed
by the normal completion path. The quality gate was re-run independently
against the submitted commit `419e17fc` before the merge, rather than trusted
from the worker's own report:
```text
bash -n scripts/*.sh syntax ok
bash scripts/orchestra_e2e_healthcheck.sh OK - all healthchecks passed, exit 0
bash scripts/test_healthcheck.sh all checks passed, exit 0
```
Worker state is empty: no sessions, no leases, no release transactions, and
`tmux -L orchestra ls` lists nothing. The two panes the previous section left
for an operator were already gone by then.
| Half | Revision |
|---|---|
| Coordinator, homesrv container | `d6ee10f` |
| Worker, workpc systemd | `d6ee10f` |
```text
commit d6ee10f02843e2267c859b03d17db00995941e64
coordinator sha256 f8a847477ae0ac03fe78338775f52ed1f167f0e67eab557feffe4d67da3ad99e
worker sha256 fe7d2188b12369fb7682d3424fa11e2db47abed9c55313291a4b467b5972d02d
```
### Still open after run 12
- **29 blocked `test-e2e` tasks**, mostly burn-in debris. Operator hygiene, not
lifecycle code.
- **Three queued `correx` tasks** that cannot be scheduled, because `correx` has
no entry in the coordinator's `config.jsonc`. Either give it one or block them
explicitly.
- **Gitea returns 422 for a review on your own pull request**, so
`REQUEST_CHANGES` needs a separate bot account for `ORCHESTRA_GITEA_TOKEN`.
- **A blocked task that is never reopened can still loop** on a dead release
transaction. F59 covers failed, not blocked.
## Run 13, 2026-08-28: F60, and the ring proven under a real failure
### F60: a transaction must settle deterministically, in every case
`3c7cf95`. F58 and F59 each fixed one case of the same rule. F60 states the
rule and covers the rest:
> a transaction that can no longer be legitimately committed must be abandoned
> deterministically, and must never spin on an answer that cannot change.
- **Terminal is failed or completed.** Both drop the transaction and free the
session. Completed was the gap F59 left.
- **Blocked keeps the transaction**, because a reopen returns the task to the
queue and that exact owner can still commit.
- **`TaskBlocked` now retains the ending epoch**, as `TaskReleased` already did.
Without it a reopened task has no epoch for the late-handoff path to fence
against, so keeping the transaction would be a lie.
- **A refusal parks the commit**, 30s backing off to a 5 minute cap. Any event
about the task un-parks it. A reopen arrives as `TaskCorrected`, so the rule
cannot be a list of event types.
- **A transport failure is not an answer** and retries at once. The park keys on
a 4xx `StatusError`, never on any error.
### Proven live, and the contrast with run 10
Task `06G4KVSHAK9B8M9K8HENAF23CG`, same race-guard rig: force the expiry inside
the push, then hand the task to `race-guard-probe`.
```text
19:54:10.888 transaction 06G4KVYT91GQH5ZYY6CNG6FWZ8 opens at prepared
19:54:10.907 forced expiry accepted
19:54:10.916 successor lease to race-guard-probe accepted
19:54:11.776 commit refused once, 409 lease not owned, parked
19:54:15.741 superseded by lease 06G4KVYSXJH053K3DAG6FKHD20, transaction abandoned
```
Run 10's equivalent ran roughly 5,000 retries over seven hours and needed an
operator to clear the state file by hand. This one asked once and settled
itself in four seconds. Worker state afterwards: no sessions, no leases, no
transactions, and `tmux -L orchestra ls` empty.
### F18 populated under a real failure
The gap the previous section recorded is closed. The same run produced this
health payload, two distinct observations rather than one overwritten slot:
```json
"observations": [
{"message": "release 06G4KVSHAK9B8M9K8HENAF23CG commit: federation: 409 Conflict: lease not owned",
"count": 1, "first": "2026-08-28T19:54:11.776815524Z", "last": "2026-08-28T19:54:11.776815524Z"},
{"message": "release 06G4KVSHAK9B8M9K8HENAF23CG superseded by lease 06G4KVYSXJH053K3DAG6FKHD20: abandoning transaction 06G4KVYT91GQH5ZYY6CNG6FWZ8",
"count": 1, "first": "2026-08-28T19:54:15.74116014Z", "last": "2026-08-28T19:54:15.74116014Z"}
]
```
Under the old single slot the abandonment message would have erased the 409
that caused it, and the causal chain would have been unreadable. Both counts
are 1, which is itself the F60 evidence: nothing looped.
### Deployed state
| Half | Revision |
|---|---|
| Coordinator, homesrv container | `3c7cf95` |
| Worker, workpc systemd | `3c7cf95` |
```text
commit 3c7cf95d8cdaaff5dcec460c5d7874f7f4f0dbea
coordinator sha256 3fec1e99a638da851f5c970abaf80a006dc87425936f71fe33f10830f4bac573
worker sha256 a373445f167f0a321dfc3866a1d2ed703b670b0fb3c188a8d00ad8099586dfdf
```
`orchestra-f18-baseline` tags `d6ee10f`, the post-F18 deployed baseline, and is
pushed. The branch is pushed too: 91 commits, because nothing had gone up since
`97a9c65` on 2026-07-31.
### Still open
Unchanged and all operational rather than runtime: 29 blocked `test-e2e` tasks,
three unschedulable `correx` tasks, and the Gitea 422 on reviewing your own
pull request. The rig task above will requeue when the probe lease expires and
start fresh, which is correct: its anchor was abandoned, not committed.
## Run 14, 2026-08-28: plan-phase execution proven on `3c7cf95`
Four tasks. Two carried a full lifecycle to a pull request, one failed for a
cause outside Orchestra, and the F60 rig task failed on its retry budget.
### The rung: the worker runs the sealed plan's commands
Task `06G4M8WHGQ4P3GQMPEEH0RJRHM`, plan `789ed6a8477b`, both phases verified
against the plan document rather than the request.
```text
plan phase-1 - run: ["bash", "-n", "scripts/orchestra_e2e_healthcheck.sh"]
- run: ["bash", "scripts/orchestra_e2e_healthcheck.sh"]
event v19 commands identical, exit_codes [0,0], at_sha 3b66b2b6a669,
evidence_ref a5dbc4ecc9ce, status awaiting_manual_verification
plan phase-2 - run: ["bash", "-n", "scripts/orchestra_e2e_healthcheck.sh"]
- run: ["bash", "-n", "scripts/test_healthcheck.sh"]
- run: ["bash", "scripts/orchestra_e2e_healthcheck.sh"]
event v20 commands identical, exit_codes [0,0,0], same at_sha and plan_ref
```
Every record binds `plan_ref`, `phase_id`, `at_sha`, `evidence_ref`,
`lease_epoch` and `harness_id`. Rotation ran between every work phase with
`TaskPickupValidated` each time.
**Not yet proven:** a successor picking up mid-implement. Both phases verified
under one lease epoch, so no rotation separated phase 1 from phase 2.
**The planner found the legal route to a forbidden command.**
`scripts/test_healthcheck.sh` may not be executed, but `["bash", "-n", ...]`
covers any path, so it syntax-checked the file it could not run.
### The refusal loop is actionable, not merely observable
Task `06G4M6HF1Z3EREX1X3NEKSHP24` sealed a plan naming
`bash scripts/test_healthcheck.sh`, which the policy forbids.
```text
20:45:02 phase request refused: "bash scripts/test_healthcheck.sh" is not in the policy
20:45:33 implement <- the planner rewrote and resealed, 31 seconds later
20:48:03 in_review
```
F52 refused on the planner, F55's answer path delivered the reason, and the
agent corrected. Count stayed at 1 for both tasks that hit it. A brief that
states the policy avoids the round trip entirely: the `--list-checks` task was
told the allowed argv and sealed on the first attempt.
### F61, recorded and deliberately not built
The verification allow-list is enforced and never stated to the planner. Every
plan that guesses pays one refused round trip. It is not a one-liner:
`agentctx.Input.Policy` is filled from the worker's `SafeOperations`, while the
verification policy is coordinator-side by design, so this needs a new channel.
The reason to promote it later is local models. A strong planner consumed the
refusal and repaired the plan in 31 seconds. A weaker one may propose forbidden
commands repeatedly, because it cannot infer the allowed substitute. Then F61
stops being latency and becomes model compatibility.
### opencode is not running, and the reason was only in the pane
Task `06G4M6HH5BAM235AC4PWYXF1HM` failed at attempt 3 having never left
`frame`. Its worktree held only `launch.md`. The pane said why. The blocked
thing is the selected model, not the OpenCode Zen free tier, which works:
```text
This model is not available in your country.
▣ Build · Muse Spark 1.2 Free OpenCode Zen
```
Orchestra behaved correctly throughout: lease, no progress, expire, requeue,
fail. Two things are worth keeping.
- **The adapter cannot resolve a session file for opencode**, so activity is
`unknown` and the worker falls back to agent status. The failure therefore
reads as "done and pane unchanged" rather than "dead". Orchestra cannot tell
finished from never-started on this harness.
- **No observation carried the cause.** Four of them named the lease, the
rotation and the activity. The fatal line existed only in the pane capture,
which Orchestra publishes but nothing summarises. Parsing harness chrome is
not Orchestra's job; surfacing the capture is.
**Do not route to `workpc-opencode` until its model is reachable.** Every task
sent there burns three attempts and then fails. `~/.config/opencode/opencode.jsonc`
sets no top-level `model`, so OpenCode picks its own default. The file already
declares reachable `nvidia-nim` and `llama-cpp` providers.
### The retry budget counts idleness, not just failure
The F60 rig task `06G4KVSHAK9B8M9K8HENAF23CG` failed at attempt 3 mid-review.
Two attempts were rig-induced expiries and the third was a natural idle
expiry. Every lease expiry increments `attempt`, and an agent going quiet at a
phase boundary is a routine expiry here. A task making real progress can spend
its retry budget on idleness. The two clean tasks never expired at all, so this
is a dynamic to watch rather than a defect to fix.
F59 was confirmed live twice more: both failed tasks left no release
transaction and no session behind, on either worker.
### Result
| Task | Harness | Outcome |
|---|---|---|
| `06G4M6HF1Z3EREX1X3NEKSHP24` | workpc-claude | in_review, recovered from a policy refusal |
| `06G4M8WHGQ4P3GQMPEEH0RJRHM` | workpc-claude | in_review, plan-phase execution proven |
| `06G4M6HH5BAM235AC4PWYXF1HM` | workpc-opencode | failed, model unavailable |
| `06G4KVSHAK9B8M9K8HENAF23CG` | workpc-claude | failed, retry budget |
Both workers hold no sessions and no release transactions. Deployed pair is
still `3c7cf95` on both halves.
+410
View File
@@ -0,0 +1,410 @@
# Debt ledger: design from what the tree already supports
Written 2026-08-29 against `3c7cf95`. Read with `BURNIN.md` (the run ledger),
`PLAN-SPEC-DESIGN.md` and `AUDIT.md`.
This is a design, not an implementation. Nothing here has been built.
## The problem being solved
AI raises implementation throughput faster than it raises integration, cleanup,
observability, consolidation and architectural understanding. The residue is
fixes, compatibility paths, operational workarounds, duplicate config, adapter
gaps and one-off patches. Some are justified. Some become permanent because
nobody remembers why they were added or what they have cost since.
The target is a first-class debt ledger driven by evidence from real tasks, not
a second TODO list.
## Classes
Four, and no more. A fifth class invented at runtime makes the priority model
meaningless.
| Class | Meaning |
|---|---|
| correctness | Known behaviour is wrong or violates an invariant. |
| operational | The system works, but diagnosis, recovery, deployment, observability or operation repeatedly costs time. |
| structural | Duplication or architecture is demonstrably raising the cost of future changes. |
| polish | Cleanup or consistency work with no demonstrated cost yet. |
Correctness and operational debt gain priority quickly. Structural debt needs
evidence that it is causing repeated work. Polish never becomes work on its own.
## 1. What Orchestra already has
The event log is the durable spine, and it already carries most of what a debt
ledger needs. Every row below is mechanically countable today, with no new
instrumentation.
| Evidence | Source | Carries |
|---|---|---|
| Failure shape | `TaskFailed`, `TaskBlocked` | 12 typed `block_reason` values, 6 `failure_class` values |
| Retry cost | `Task.Attempt`, `NextRetryAt`, `FailureClass` | how many leases a task burned, durable in the projection |
| Review findings | `ReviewRecorded` | `{ID, Severity, File, Line, Claim, Evidence}`, bound to a `ResultSHA` |
| Out-of-scope discoveries | `DeferredFindingRecorded` | `{Summary, Why}`, already recorded outside agent context |
| Plan contradicted by code | `PlanMismatchRecorded` | `{PlanRef, PhaseID, AtSHA, Observed, Contradicts, Evidence[]}` |
| Verification history | `PlanPhaseVerified` | `{Commands, ExitCodes, AtSHA, EvidenceRef}` per phase |
| Rework rounds | `TaskChangesRequested`, `TaskSubmitted` | how many times a change went back |
| Provenance | `TaskCreated` | `Source`, `ExternalID`, `Parent` |
Three existing patterns matter more than the data.
**Global-subject events already work.** `QuotaReported` and `StandupAdvisory`
use `TaskID: "system"` and are whitelisted in `internal/domain/domain.go:279`.
A debt item needs no new subject mechanism.
**`GenerateStandupAdvisory` is the precedent for the shape.** It is a
persisted, read-only recommendation, and applying it is a separate
approval-gated operation. A debt ledger is that pattern with better inputs.
**`CheckSubmission` is the precedent for eligibility.** It is a pure function
of task, commit and gate run, returning `{Eligible bool, Reasons []string}`.
Debt eligibility should have the same shape.
## 2. What evidence is missing
**Worker observations are not durable.** This is the largest gap. The F18 ring
lives in worker memory and reaches the coordinator inside `WorkerHealth` on
heartbeat. `Registry.persistedState` holds captures, commands and workers only,
with no health. A coordinator restart erases every observation.
Operational debt is exactly what that ring holds. Repeated 409s, unrenewed
leases, adapter gaps.
**No operator-intervention record.** Every manual repair in run 12 and run 13
left no trace in Orchestra. State-file edits, worker restarts, two manual
transaction cleanups. "Required manual recovery" is the strongest priority
signal available, and it is currently unrecorded.
**No diagnosis-cost signal.** Time spent diagnosing is not measured. The
closest proxy is wall time in `blocked` or `needs_attention`.
**No component dimension.** Findings carry file paths. Tasks carry none.
Breadth across components has to be derived from paths.
**No link from a repair commit back to what it repaired.** The Fxx-to-commit
mapping exists only in `BURNIN.md` prose.
**Deferred findings are too thin.** `{Summary, Why}` has no class, no severity,
no paths and no evidence refs.
## 3. Minimal durable data model
No new subsystem. Two event types on the existing spine, one projection beside
`Task`.
```go
// DebtObservation is one piece of counted evidence, always pointing at the
// event that produced it. Provenance is an event id, never prose.
type DebtObservation struct {
// Exactly one of EventID and LegacyRef. See section 8.
EventID string `json:"event_id,omitempty"`
LegacyRef string `json:"legacy_ref,omitempty"`
TaskID string `json:"task_id,omitempty"`
Kind string `json:"kind"` // block_reason, failure_class, review_finding,
// plan_mismatch, manual_intervention, deferred_finding
Signature string `json:"signature"` // the mechanical dedup key, see section 4
Detail string `json:"detail"`
Paths []string `json:"paths,omitempty"`
At time.Time `json:"at"`
}
type DebtItem struct {
ID string `json:"id"` // continues the Fxx namespace
Class DebtClass `json:"class"` // correctness, operational, structural, polish
Status DebtStatus `json:"status"` // observed, eligible, scheduled, repaired, withdrawn
Symptom string `json:"symptom"`
Consequence string `json:"consequence"`
Severity Severity `json:"severity"` // reuse review.Severity
Paths []string `json:"paths,omitempty"`
Signatures []string `json:"signatures"` // every key that attaches here
Observations []DebtObservation `json:"observations"`
IntroducedIn string `json:"introduced_in,omitempty"` // task id or commit
RepairBoundary string `json:"repair_boundary,omitempty"`
RepairTask string `json:"repair_task,omitempty"`
RepairCommit string `json:"repair_commit,omitempty"`
}
```
Every counted field the priority model needs is derived, never stored:
```text
recurrence len(Observations)
blocked_tasks distinct TaskID where Kind is block_reason
manual_interventions count of Kind == manual_intervention
breadth distinct components derived from Paths
```
A stored count invites drift from the log. A derived one cannot drift.
Two events carry it. `DebtObserved` appends one observation. `DebtItemUpdated`
records class, status, severity or a merge. Both need adding to the `allowed`
map and a validator, which is the work every existing event type already did.
## 4. Deduplication without corrupting provenance
**A signature is computed from typed facts, never from prose.**
```text
operational class + block_reason + harness + normalized component
correctness class + failure_class + normalized component
structural class + finding severity + normalized component
```
Exact signature match attaches automatically. That is the only automatic path.
**Signatures are versioned, in the string itself.**
```text
v1:operational:lease_not_renewed:workpc-opencode:federation
```
Normalization rules will change. Without a version, changing them silently
regroups every historical observation, and the recurrence counts that drive
eligibility move underneath the items that already used them. A `v2` signature
never matches a `v1` one, so old evidence keeps the grouping it was counted
under.
**A model may propose, never merge.** Clustering emits `DebtMergeSuggested`
carrying both item ids and its reasoning. The merge is an operator action or a
policy threshold, recorded as `DebtItemUpdated`.
**Merges are additive.** The surviving item gains the other's signatures and
observations. The absorbed item becomes `withdrawn` with a pointer, and its
observations keep their original event ids. Nothing is rewritten, so a bad
merge is reversible by reading the log.
**Every observation names its event.** A debt item's evidence is always
checkable against the log that produced it. That rule is what stops a fuzzy
proposal from becoming an unverifiable claim.
## 5. Eligibility policy
A pure function, mirroring `CheckSubmission`.
```go
func CheckDebtEligibility(item DebtItem) DebtCheck // {Eligible bool, Reasons []string}
```
| Class | Becomes eligible when |
|---|---|
| correctness | first confirmed observation, always |
| operational | recurrence >= 3 across >= 2 distinct tasks, or >= 1 manual intervention |
| structural | >= 3 review findings, or >= 2 tasks blocked or reworked in the same component |
| polish | never automatically, operator promotion only |
`Reasons` lists what fired and what did not. "Not eligible" alone sends an
operator reading code, which is the mistake `SubmissionCheck` already documents.
Eligible means a task may be created. It does not create one.
## 6. How maintenance and consolidation fit the existing lifecycle
A maintenance task is an ordinary task. Source `debt`, external id the debt item
id, phases as usual.
```text
research → plan → implement → verification → review → submission
```
Nothing in the lifecycle changes. One extension is needed: the task's brief must
carry the debt item's evidence. That is the mechanism research and plan
artifacts already use, a CAS ref on the task.
A consolidation task is the same thing with a research brief that asks:
- which temporary paths are still necessary?
- which fixes now duplicate each other?
- which compatibility branches are obsolete?
- which config or state is represented twice?
- which abstractions exist only because of defects that have since disappeared?
- what can now be deleted safely?
Its first output is a deletion plan, never an automatic refactor. Verification
uses the project's existing policy. Review is the normal independent review.
**The repair gets no discount.** Classification as debt changes what gets
scheduled, never what gets checked.
### Success metrics for maintenance work
Lines of code are not a metric. Prefer evidence that can be checked:
- behaviour preserved
- tests preserved or strengthened
- branches removed
- obsolete types removed
- compatibility code removed
- duplicate config removed
- manual recovery paths eliminated
## 7. What must not be automated
- No autonomous cleanup agent, and no repo-wide sweep.
- No LLM score treated as authoritative. Counted facts decide, models propose.
- No automatic merge of debt items.
- No maintenance task from polish without an operator.
- No deletion without an independent review.
- No auto-closing an item because a commit touched the file. Closure needs a
repair task or a stated run of clean evidence.
- No new classes invented at runtime.
## 8. Migration, so there is one track and not two
`BURNIN.md` holds 58 Fxx entries in the form `| F52 | commit | prose |`.
`AUDIT.md` holds the older narrative.
One-time import, by hand, with a class assigned per row. Each becomes a
`DebtItem` with `Status: repaired`, its commit as `RepairCommit`, and one
observation citing the ledger.
**Imported observations carry legacy provenance, never a fabricated event id.**
The Fxx history predates the capability that would have produced an event, and
inventing one would break the provenance rule the ledger exists to enforce.
```go
EventID string `json:"event_id,omitempty"`
LegacyRef string `json:"legacy_ref,omitempty"` // "BURNIN.md:F18"
```
Exactly one is required. A reader can then tell a counted fact from an imported
claim at a glance, and the counts that drive eligibility can exclude imported
evidence if that turns out to matter.
**Keep the namespace.** New items continue at F62. Two numbering schemes would
be the first structural debt the ledger itself creates.
After import, `BURNIN.md` stops being the item ledger and remains the run
narrative. The defect table becomes generated output from the projection.
### The history that tests the model
These are not special cases. They are the check on whether the model can
represent the history this project already has.
| History | Expected class | Why |
|---|---|---|
| F18, the single `last_error` slot | operational | Repeatedly destroyed causal evidence before it was fixed. |
| F43 to F46, the submission path | correctness | Apparently implemented, actually unreachable. |
| F61, planner learns policy by refusal | operational | Costs a round trip per plan, and becomes model compatibility on weaker planners. |
| Duplicated `quality_gate` config | structural | Only if it keeps causing drift or operator mistakes. |
| Old blocked burn-in tasks | neither | Hygiene, unless one exposes a runtime defect. |
| The blocked release loop, F57 to F60 | correctness and operational | Wrong behaviour, and it required manual cleanup twice. |
## 9. Smallest slice to live-prove first
**A read-only projection over the existing log. No new events, no writes, no
schema commitment.**
```text
internal/domain/debt.go DebtItem, DebtObservation, signature, classification
internal/store/debt_projection.go the fold over s.Events(0), and the gap report
internal/operations/debt.go CheckDebtEligibility, and later the actions
GET /v1/debt read-only, tui surface
```
The projected type stays out of `operations`. Baking a read model into the
command layer in the first slice is the mistake that would be hardest to undo
later. `operations` owns actions: `CheckDebtEligibility` now, and
`SuggestDebtMerge`, `PromoteDebtItem` and `ScheduleDebtRepair` when they exist.
The proof is what the projection can and cannot recover from canonical history.
Section 2 already says worker observations are not durable and operator
interventions are unrecorded, so demanding their reconstruction would be asking
the projection to invent evidence.
The first burn-in is therefore three requirements, not one:
1. Recover every debt signal that canonical history actually encodes.
2. Report, explicitly and per kind, where known debt cannot be reconstructed.
3. Never infer a missing observation from `BURNIN.md`.
Incompleteness is part of the result, not a failure. The 409 loop lived in the
F18 worker ring rather than in an event, so the ledger should say it has no
durable evidence for that shape. That statement is what makes slice two
necessary, measurably rather than by assertion.
What the projection should recover from the log alone:
- repeated `lease_expired` and `retry_limit` on one harness, which is the
opencode failure shape across four tasks
- the retry-idleness dynamic, attached to the tasks that expired
- review findings grouped by component, if any repeat
A model that cannot represent debt already known is wrong, and no schema has
been committed to yet. That is the cheapest place to find out.
**Slice two** makes worker observations durable, because that is the input the
projection will most visibly lack.
**Slice three** adds `DebtObserved` and manual-intervention recording.
## Scope boundary
This design changes no part of the task lifecycle, phases, leases, review,
submission, federation or the plan machinery. The only extensions it needs are
two event types, one projection, one pure eligibility function and one
read-only endpoint.
---
# Slice one, run against real history
Built and deployed as `a757cff`. `GET /v1/debt` folded 881 events from the
live log, wrote nothing, and produced 15 candidate items and 3 gaps.
## What it recovered
```text
v1:operational:lease_expired:workpc-opencode:lease r=41 tasks=4
v1:operational:lease_expired:workpc-claude:lease r=29 tasks=13
v1:operational:lease_failure:-:lease r=20 tasks=16
v1:operational:system_error:-:lease r=11 tasks=7
v1:correctness:handoff_validation:-:lease r=5 tasks=5
v1:correctness:plan_mismatch:-:... r=1 tasks=1
v1:structural:minor:-:scripts/orchestra_e2e_healthcheck.sh r=1 tasks=1
```
The opencode failure shape is the top item, found mechanically. Run 14
diagnosed the same thing by hand from a pane capture. The retry-idleness
dynamic is the second item, and it is now a number: 29 expiries across 13
tasks on one harness.
Seven items are eligible under the stated policy. Polish and the single
structural finding correctly are not.
## What it reported that it could not see
```text
durable=false manual_intervention no event type records an operator repair
durable=false worker_observation worker health is not persisted
durable=true deferred_finding carried by the log, this history has none
```
The 409 release loop does not appear, and it should not. That evidence lived in
the F18 worker ring, which no event carries. The ledger says so rather than
inferring it, which is the result slice two exists to change.
## Four defects the first run exposed
**The component part is too coarse for lease evidence.** Everything
lease-related normalizes to `lease`, so `lease_expired:workpc-claude` is one
bucket holding idle agents, rig interference and real failures. Recurrence 29
is true and the item is not a defect.
**`harness` is often empty on block reasons.** `TaskBlocked` payloads do not
always carry `harness_id`, so `lease_failure:-` mixes harnesses that should be
separate items.
**Path normalization mangled a mismatch reference.**
`scripts/orchestra_e2e_healthcheck.sh:12` became
`scripts/orchestra_e2e_healthcheck.sh_12`, because the signature sanitizer
replaces `:` and the mismatch evidence field holds prose, not clean paths.
**Recurrence alone is the wrong sort.** 41 occurrences on 4 tasks currently
outranks 29 on 13 tasks. Breadth is in the design and not yet in the ordering,
which is the priority function slice one deliberately omitted.
None of these required a schema commitment to discover. That was the point of
making the first slice read-only.
+43 -6
View File
@@ -157,6 +157,22 @@ type releaseTransaction struct {
AgentReleased bool `json:"agent_released,omitempty"`
LastError string `json:"last_error,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
// NextAttemptAt parks a commit the coordinator has refused. A refusal is
// an answer about the task, not a transport failure, so it stays true
// until something about the task changes.
NextAttemptAt time.Time `json:"next_attempt_at,omitempty"`
Attempts int `json:"attempts,omitempty"`
}
// releaseBackoff spaces out refused commits. The first wait is long enough
// that a parked transaction stops filling the observation ring, and the cap
// keeps a reopen from waiting more than five minutes to be noticed.
func releaseBackoff(attempts int) time.Duration {
d := 30 * time.Second << (attempts - 1)
if attempts < 1 || d > 5*time.Minute {
return 5 * time.Minute
}
return d
}
type projectConfig struct {
Repo string `json:"repo"`
@@ -894,18 +910,31 @@ func (w *worker) advanceRelease(ctx context.Context, id string, s herdr.Session)
w.releases[id] = tx
_ = w.save()
}
if tx.Phase == "anchor_pushed" && time.Now().Before(tx.NextAttemptAt) {
return
}
if tx.Phase == "anchor_pushed" {
// The epoch comes from the transaction, not from w.leases: an expiry
// replay deletes the lease, and the coordinator needs the epoch of the
// lease this anchor was pushed under to accept the late commit.
if err := w.api.Release(ctx, id, tx.Ref, tx.AnchorSHA, tx.ID, tx.LeaseEpoch, tx.LeaseVersion, w.sessionEvidence(ctx, id, s)); err != nil {
tx.LastError, tx.UpdatedAt = err.Error(), time.Now().UTC()
// A refusal is the coordinator's answer about who owns the task.
// It cannot change until an event about that task does, so asking
// again every five seconds only burns the observation ring. A
// transport failure is the opposite and must retry at once.
var refused *federation.StatusError
if errors.As(err, &refused) && refused.Code >= 400 && refused.Code < 500 {
tx.Attempts++
tx.NextAttemptAt = time.Now().UTC().Add(releaseBackoff(tx.Attempts))
}
w.releases[id] = tx
_ = w.save()
w.recordError(fmt.Errorf("release %s commit: %w", id, err))
return
}
tx.Phase, tx.LastError, tx.UpdatedAt = "event_committed", "", time.Now().UTC()
tx.NextAttemptAt, tx.Attempts = time.Time{}, 0
w.releases[id] = tx
_ = w.save()
}
@@ -1313,6 +1342,13 @@ func (w *worker) once(ctx context.Context) error {
if t, ok := created(e); ok {
w.tasks[t.ID] = t
}
// Any event about this task is the change a parked commit was waiting
// for. A reopen arrives as TaskCorrected, so this cannot be a list of
// specific types without going stale.
if tx, parked := w.releases[e.TaskID]; parked && !tx.NextAttemptAt.IsZero() {
tx.NextAttemptAt, tx.Attempts = time.Time{}, 0
w.releases[e.TaskID] = tx
}
if e.Type == "TaskLeased" {
var p struct {
HarnessID string `json:"harness_id"`
@@ -1403,7 +1439,7 @@ func (w *worker) once(ctx context.Context) error {
w.leases[e.TaskID] = l
}
}
if e.Type == "TaskReleased" || e.Type == "TaskFailed" || e.Type == "TaskBlocked" {
if e.Type == "TaskReleased" || e.Type == "TaskFailed" || e.Type == "TaskBlocked" || e.Type == "TaskCompleted" {
if e.Type == "TaskReleased" {
var p struct {
TransactionID string `json:"transaction_id"`
@@ -1425,12 +1461,13 @@ func (w *worker) once(ctx context.Context) error {
// mapping protects nothing. F30: a transaction stuck at "prepared"
// held the session forever once its pane was gone, health() kept
// reporting ActiveTask, and the harness never leased again.
// A failed task is terminal: no successor will ever lease it, so
// its anchor protects nothing and its transaction can only retry
// a refusal forever. Blocked is different, because a reopen still
// produces a successor.
// Failed and completed are terminal: no successor will ever lease
// the task, so the anchor protects nothing and the transaction can
// only retry a refusal forever. Blocked is different, because a
// reopen returns the task to the queue and the epoch that ended is
// still on record, so that exact commit can still be accepted.
tx, releasing := w.releases[e.TaskID]
if !releasing || tx.Ref == "" || e.Type == "TaskFailed" {
if !releasing || tx.Ref == "" || e.Type == "TaskFailed" || e.Type == "TaskCompleted" {
if releasing {
delete(w.releases, e.TaskID)
}
+117
View File
@@ -1327,3 +1327,120 @@ func TestObservationRingEvictsLeastRecentlySeen(t *testing.T) {
t.Fatal("least recently seen entry survived")
}
}
// F60. A refusal is an answer about the task, not a transport failure, and it
// stays true until something about that task changes. Run 10's blocked task
// asked 5,000 times over seven hours and got the same 409 every time.
func TestRefusedCommitParksUntilSomethingChanges(t *testing.T) {
var commits int
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/handoff"):
commits++
http.Error(w, "lease not owned", http.StatusConflict)
case strings.HasSuffix(r.URL.Path, "/events"):
_, _ = w.Write([]byte(`{"cursor":0,"events":[{"seq":1,"id":"c","type":"TaskCorrected","task_id":"t","version":9,"payload":{"state":"queued"},"surface":"web"}]}`))
default:
w.WriteHeader(http.StatusNoContent)
}
}))
defer s.Close()
tx := releaseTransaction{ID: "tx", Phase: "anchor_pushed", Ref: "sha256:abc", AnchorSHA: "abc", LeaseEpoch: "e1"}
w := &worker{
api: federation.Client{BaseURL: s.URL, WorkerID: "h", Token: "t"},
harnessID: "h",
tasks: map[string]domain.Task{"t": {ID: "t"}},
sessions: map[string]herdr.Session{"t": {PaneID: "pane"}},
leases: map[string]lease{},
releases: map[string]releaseTransaction{"t": tx},
quarantined: map[string]bool{},
statePath: t.TempDir() + "/state.json",
hard: .75,
}
w.advanceRelease(context.Background(), "t", w.sessions["t"])
w.advanceRelease(context.Background(), "t", w.sessions["t"])
if commits != 1 {
t.Fatalf("refused commit retried %d times without waiting", commits)
}
if w.releases["t"].NextAttemptAt.IsZero() {
t.Fatal("refused commit was not parked")
}
// A reopen arrives as TaskCorrected. Any event about the task is the
// change the parked commit was waiting for, so the same tick retries it
// and, still refused, parks it again.
if err := w.once(context.Background()); err != nil {
t.Fatal(err)
}
if commits != 2 {
t.Fatalf("an event about the task did not un-park its commit, commits=%d", commits)
}
if w.releases["t"].NextAttemptAt.IsZero() {
t.Fatal("the second refusal did not park it again")
}
}
// The opposite case, and the one a backoff must not break: the coordinator is
// unreachable or broken rather than answering. That says nothing about who
// owns the task, so it has to retry at once.
func TestTransientCommitFailureKeepsRetryingAtOnce(t *testing.T) {
var commits int
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/handoff") {
commits++
http.Error(w, "upstream unavailable", http.StatusBadGateway)
return
}
w.WriteHeader(http.StatusNoContent)
}))
defer s.Close()
w := &worker{
api: federation.Client{BaseURL: s.URL, WorkerID: "h", Token: "t"},
harnessID: "h",
tasks: map[string]domain.Task{"t": {ID: "t"}},
sessions: map[string]herdr.Session{"t": {PaneID: "pane"}},
leases: map[string]lease{},
releases: map[string]releaseTransaction{"t": {ID: "tx", Phase: "anchor_pushed", Ref: "sha256:abc", AnchorSHA: "abc", LeaseEpoch: "e1"}},
quarantined: map[string]bool{},
statePath: t.TempDir() + "/state.json",
hard: .75,
}
w.advanceRelease(context.Background(), "t", w.sessions["t"])
w.advanceRelease(context.Background(), "t", w.sessions["t"])
if commits != 2 {
t.Fatalf("transport failure was parked like a refusal, commits=%d", commits)
}
if !w.releases["t"].NextAttemptAt.IsZero() {
t.Fatal("transport failure must not park the transaction")
}
}
// Completion is terminal for a release transaction just as failure is. The
// task is done; nothing will ever lease it again to pick the anchor up.
func TestCompletedTaskDropsItsReleaseTransaction(t *testing.T) {
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/events") {
_, _ = w.Write([]byte(`{"cursor":0,"events":[{"seq":1,"id":"c","type":"TaskCompleted","task_id":"t","version":9,"payload":{"report_ref":"sha256:r"},"surface":"system"}]}`))
return
}
w.WriteHeader(http.StatusNoContent)
}))
defer s.Close()
w := &worker{
api: federation.Client{BaseURL: s.URL, WorkerID: "h", Token: "t"},
harnessID: "h",
backend: deadTmuxBackend(t),
tasks: map[string]domain.Task{"t": {ID: "t"}},
sessions: map[string]herdr.Session{"t": {PaneID: "pane"}},
leases: map[string]lease{},
releases: map[string]releaseTransaction{"t": {ID: "tx", Phase: "anchor_pushed", Ref: "sha256:abc", AnchorSHA: "abc", LeaseEpoch: "e1"}},
quarantined: map[string]bool{},
statePath: t.TempDir() + "/state.json",
hard: .75,
}
if err := w.once(context.Background()); err != nil {
t.Fatal(err)
}
if len(w.releases) != 0 || len(w.sessions) != 0 {
t.Fatalf("completed task kept its release: releases=%v sessions=%v", w.releases, w.sessions)
}
}
+11
View File
@@ -445,6 +445,17 @@ func main() {
}
json.NewEncoder(w).Encode(s.Events(n))
})
mux.HandleFunc("/v1/debt", func(w http.ResponseWriter, r *http.Request) {
// Read-only, and deliberately so. This projection is evidence about
// history, not a new kind of truth: nothing here writes an event,
// schedules work, or decides that something is worth repairing.
ledger := s.DebtLedger()
out := struct {
domain.DebtLedger
Eligible []operations.DebtCandidate `json:"eligible"`
}{ledger, operations.EligibleDebt(ledger)}
json.NewEncoder(w).Encode(out)
})
mux.HandleFunc("/v1/handoffs", func(w http.ResponseWriter, r *http.Request) {
out := make([]domain.Event, 0)
for _, e := range s.Events(0) {
+260
View File
@@ -0,0 +1,260 @@
package domain
import (
"fmt"
"strings"
"time"
"orchestra/internal/review"
)
// Debt is the projected read model for accumulated cost: the defects,
// workarounds and operational burdens that real tasks keep paying for. It is
// derived from the event log rather than written by hand, so a claim in the
// ledger can always be checked against the events that produced it.
//
// Four classes, and no more. A fifth invented at runtime makes the priority
// model meaningless, because the thresholds in operations.CheckDebtEligibility
// are stated per class.
type DebtClass string
const (
// DebtCorrectness is behaviour that is wrong or violates an invariant.
DebtCorrectness DebtClass = "correctness"
// DebtOperational works, but repeatedly costs time to diagnose, recover,
// deploy, observe or operate.
DebtOperational DebtClass = "operational"
// DebtStructural is duplication or architecture demonstrably raising the
// cost of future changes.
DebtStructural DebtClass = "structural"
// DebtPolish has no demonstrated cost yet, and never promotes itself.
DebtPolish DebtClass = "polish"
)
func (c DebtClass) Valid() bool {
switch c {
case DebtCorrectness, DebtOperational, DebtStructural, DebtPolish:
return true
}
return false
}
type DebtStatus string
const (
DebtObserved DebtStatus = "observed"
DebtEligible DebtStatus = "eligible"
DebtScheduled DebtStatus = "scheduled"
DebtRepaired DebtStatus = "repaired"
DebtWithdrawn DebtStatus = "withdrawn"
)
// ObservationKind names where one piece of evidence came from. It is the kind
// of the source fact, never an interpretation of it.
type ObservationKind string
const (
ObservationBlockReason ObservationKind = "block_reason"
ObservationFailureClass ObservationKind = "failure_class"
ObservationReviewFinding ObservationKind = "review_finding"
ObservationPlanMismatch ObservationKind = "plan_mismatch"
ObservationDeferredFinding ObservationKind = "deferred_finding"
ObservationManualIntervention ObservationKind = "manual_intervention"
ObservationWorkerFailure ObservationKind = "worker_observation"
)
// SignatureVersion prefixes every signature this build produces. Normalization
// rules will change, and without a version a change silently regroups every
// historical observation, moving the recurrence counts that eligibility was
// already decided on. A v2 signature never matches a v1 one, so old evidence
// keeps the grouping it was counted under.
const SignatureVersion = "v1"
// DebtSignature is the mechanical dedup key. Every part is a typed fact, never
// prose, because exact signature match is the only path that attaches evidence
// without a human. An empty part becomes "-" so the arity never varies.
func DebtSignature(class DebtClass, reason, scope, component string) string {
part := func(s string) string {
s = strings.TrimSpace(strings.ToLower(s))
s = strings.ReplaceAll(s, ":", "_")
if s == "" {
return "-"
}
return s
}
return strings.Join([]string{SignatureVersion, part(string(class)), part(reason), part(scope), part(component)}, ":")
}
// DebtComponent reduces a repository path to the unit that owns it. Two
// segments is the whole rule: it keeps internal/store distinct from
// internal/herdr without splitting one package across files.
func DebtComponent(path string) string {
path = strings.TrimSpace(strings.Trim(path, "/"))
if path == "" {
return ""
}
parts := strings.Split(path, "/")
if len(parts) == 1 {
return parts[0]
}
return parts[0] + "/" + parts[1]
}
// DebtObservation is one piece of counted evidence. Provenance is required and
// is either an event id or, for history imported from a markdown ledger that
// predates this projection, an explicit legacy reference. Exactly one, because
// a fabricated event id would break the rule the ledger exists to enforce.
type DebtObservation struct {
EventID string `json:"event_id,omitempty"`
LegacyRef string `json:"legacy_ref,omitempty"`
TaskID string `json:"task_id,omitempty"`
Kind ObservationKind `json:"kind"`
Signature string `json:"signature"`
Detail string `json:"detail,omitempty"`
Paths []string `json:"paths,omitempty"`
At time.Time `json:"at"`
}
func (o DebtObservation) Validate() error {
if (o.EventID == "") == (o.LegacyRef == "") {
return fmt.Errorf("observation needs exactly one of event_id and legacy_ref")
}
if o.Kind == "" {
return fmt.Errorf("observation kind required")
}
if !strings.HasPrefix(o.Signature, SignatureVersion+":") {
return fmt.Errorf("observation signature %q is not %s", o.Signature, SignatureVersion)
}
return nil
}
// DebtItem groups observations that share a signature. Every count the
// priority model needs is derived from Observations rather than stored, so a
// count can never drift from the log that justifies it.
type DebtItem struct {
ID string `json:"id"`
Class DebtClass `json:"class"`
Status DebtStatus `json:"status"`
Symptom string `json:"symptom"`
Consequence string `json:"consequence,omitempty"`
Severity review.Severity `json:"severity,omitempty"`
Signatures []string `json:"signatures"`
Paths []string `json:"paths,omitempty"`
Observations []DebtObservation `json:"observations"`
IntroducedIn string `json:"introduced_in,omitempty"`
RepairBoundary string `json:"repair_boundary,omitempty"`
RepairTask string `json:"repair_task,omitempty"`
RepairCommit string `json:"repair_commit,omitempty"`
}
// Recurrence is how many times this shape has been seen.
func (d DebtItem) Recurrence() int { return len(d.Observations) }
// AffectedTasks is how broadly the shape has spread. Distinct tasks, because
// one task failing ten times is weaker evidence than ten tasks failing once.
func (d DebtItem) AffectedTasks() int {
seen := map[string]bool{}
for _, o := range d.Observations {
if o.TaskID != "" {
seen[o.TaskID] = true
}
}
return len(seen)
}
// BlockedTasks counts distinct tasks this shape actually stopped.
func (d DebtItem) BlockedTasks() int {
seen := map[string]bool{}
for _, o := range d.Observations {
if o.Kind == ObservationBlockReason && o.TaskID != "" {
seen[o.TaskID] = true
}
}
return len(seen)
}
// ManualInterventions counts recorded operator repairs. It reads zero on any
// log written before that recording exists, which is why the ledger reports it
// as an evidence gap rather than as an absence of operator cost.
func (d DebtItem) ManualInterventions() int {
n := 0
for _, o := range d.Observations {
if o.Kind == ObservationManualIntervention {
n++
}
}
return n
}
// Components is the breadth of the shape across the tree.
func (d DebtItem) Components() []string {
seen := map[string]bool{}
out := []string{}
for _, o := range d.Observations {
for _, p := range o.Paths {
if c := DebtComponent(p); c != "" && !seen[c] {
seen[c] = true
out = append(out, c)
}
}
}
return out
}
// Imported reports whether any evidence here came from a markdown ledger
// rather than from an event. A reader can then tell a counted fact from an
// imported claim without opening the observations.
func (d DebtItem) Imported() bool {
for _, o := range d.Observations {
if o.LegacyRef != "" {
return true
}
}
return false
}
// EvidenceGap is a debt signal the ledger knows it cannot see. Reporting the
// gap is the point: a ledger that silently omits what it cannot record reads
// as "no operator cost" when it means "operator cost is not recorded".
type EvidenceGap struct {
Kind ObservationKind `json:"kind"`
Reason string `json:"reason"`
// Durable is false when no event type carries this evidence at all. It is
// true when the log could carry it and this particular log does not.
Durable bool `json:"durable"`
}
// DebtLedger is the whole read model: what history establishes, and what it
// cannot.
type DebtLedger struct {
Items []DebtItem `json:"items"`
Gaps []EvidenceGap `json:"gaps"`
// Events is how many log entries the projection folded, so a caller can
// tell an empty ledger from an unread log.
Events int `json:"events"`
}
// DebtClassForBlockReason maps a typed block reason to a class. A reason that
// is a normal lifecycle stop rather than a cost returns false: waiting for a
// human decision is the system working, not debt.
func DebtClassForBlockReason(r BlockReason) (DebtClass, bool) {
switch r {
case BlockReasonLeaseFailure, BlockReasonWorkerOffline, BlockReasonLeaseExpired, BlockReasonSystem:
return DebtOperational, true
case BlockReasonHandoffValidation, BlockReasonPlanMismatch:
return DebtCorrectness, true
}
return "", false
}
// DebtClassForFailureClass maps a worker failure class to a class. Only the
// classes a worker actually emits are listed; an unknown one is not guessed at.
func DebtClassForFailureClass(f string) (DebtClass, bool) {
switch f {
case "retry_limit", "launch_failed", "launch_transient", "launch_uncertain", "prompt_not_submitted", "lease_expired":
return DebtOperational, true
case "invalid_handoff":
return DebtCorrectness, true
}
return "", false
}
+54
View File
@@ -0,0 +1,54 @@
package domain
import "testing"
// Normalization rules will change. A v2 signature must never match a v1 one,
// or changing them silently regroups history and moves the recurrence counts
// eligibility was already decided on.
func TestSignatureCarriesItsVersion(t *testing.T) {
sig := DebtSignature(DebtOperational, "lease_expired", "workpc-opencode", "internal/herdr")
if sig != "v1:operational:lease_expired:workpc-opencode:internal/herdr" {
t.Fatalf("signature %q", sig)
}
// Arity never varies, so a missing part cannot shift the fields left.
if got := DebtSignature(DebtPolish, "deferred_finding", "", ""); got != "v1:polish:deferred_finding:-:-" {
t.Fatalf("empty parts not padded: %q", got)
}
}
func TestComponentIsTwoSegments(t *testing.T) {
for path, want := range map[string]string{
"internal/store/store.go": "internal/store",
"internal/herdr/adapter.go": "internal/herdr",
"cmd/orchestra-worker/main.go": "cmd/orchestra-worker",
"BURNIN.md": "BURNIN.md",
"": "",
} {
if got := DebtComponent(path); got != want {
t.Fatalf("component(%q) = %q, want %q", path, got, want)
}
}
}
// Imported history predates the events that would have justified it. A
// fabricated event id would break the provenance rule the ledger enforces.
func TestObservationNeedsExactlyOneProvenance(t *testing.T) {
sig := DebtSignature(DebtOperational, "x", "", "")
both := DebtObservation{EventID: "e1", LegacyRef: "BURNIN.md:F18", Kind: ObservationBlockReason, Signature: sig}
neither := DebtObservation{Kind: ObservationBlockReason, Signature: sig}
if both.Validate() == nil || neither.Validate() == nil {
t.Fatal("exactly one of event_id and legacy_ref must be required")
}
for _, ok := range []DebtObservation{
{EventID: "e1", Kind: ObservationBlockReason, Signature: sig},
{LegacyRef: "BURNIN.md:F18", Kind: ObservationBlockReason, Signature: sig},
} {
if err := ok.Validate(); err != nil {
t.Fatalf("valid observation refused: %v", err)
}
}
stale := DebtObservation{EventID: "e1", Kind: ObservationBlockReason, Signature: "v0:operational:x:-:-"}
if stale.Validate() == nil {
t.Fatal("a signature from another version must be refused")
}
}
+81
View File
@@ -0,0 +1,81 @@
package operations
import (
"fmt"
"orchestra/internal/domain"
)
// DebtCheck is why a debt item may or may not become work. Reasons are listed
// rather than summarised, for the same reason SubmissionCheck lists them: "not
// eligible" alone sends an operator reading code.
type DebtCheck struct {
Eligible bool `json:"eligible"`
Reasons []string `json:"reasons,omitempty"`
}
// CheckDebtEligibility is the whole promotion rule, as one pure function of a
// projected item. It decides only whether a maintenance task may be created.
// It never creates one, and it never changes what that task must then pass:
// classification as debt changes what gets scheduled, never what gets checked.
//
// The thresholds differ by class on purpose. Correctness and operational debt
// have already cost something measurable. Structural debt needs evidence that
// it is causing repeated work rather than merely offending taste. Polish never
// promotes itself, or the ledger becomes a permanent cleanup generator.
func CheckDebtEligibility(item domain.DebtItem) DebtCheck {
recurrence, tasks := item.Recurrence(), item.AffectedTasks()
blocked, manual := item.BlockedTasks(), item.ManualInterventions()
switch item.Class {
case domain.DebtCorrectness:
if recurrence >= 1 {
return DebtCheck{true, []string{fmt.Sprintf("correctness debt is eligible on first confirmed observation, and has %d", recurrence)}}
}
return DebtCheck{false, []string{"no confirmed observation"}}
case domain.DebtOperational:
var why []string
if manual >= 1 {
why = append(why, fmt.Sprintf("%d manual intervention(s) recorded", manual))
}
if recurrence >= 3 && tasks >= 2 {
why = append(why, fmt.Sprintf("recurred %d times across %d tasks", recurrence, tasks))
}
if len(why) > 0 {
return DebtCheck{true, why}
}
return DebtCheck{false, []string{
fmt.Sprintf("needs 3 occurrences across 2 tasks, or 1 manual intervention; has %d across %d tasks with %d interventions", recurrence, tasks, manual),
"manual interventions are not recorded by any event type, so that count reads 0 on every current log",
}}
case domain.DebtStructural:
if recurrence >= 3 {
return DebtCheck{true, []string{fmt.Sprintf("%d review findings in this component", recurrence)}}
}
if blocked >= 2 {
return DebtCheck{true, []string{fmt.Sprintf("blocked %d distinct tasks in this component", blocked)}}
}
return DebtCheck{false, []string{
fmt.Sprintf("needs 3 review findings or 2 blocked tasks in one component; has %d findings and %d blocked", recurrence, blocked),
}}
case domain.DebtPolish:
return DebtCheck{false, []string{"polish never promotes itself, an operator promotes it explicitly"}}
}
return DebtCheck{false, []string{"unknown debt class " + string(item.Class)}}
}
// EligibleDebt filters a projected ledger to what policy would allow to become
// work. It returns the check alongside each item so the reasons stay visible.
func EligibleDebt(ledger domain.DebtLedger) []DebtCandidate {
out := []DebtCandidate{}
for _, item := range ledger.Items {
if check := CheckDebtEligibility(item); check.Eligible {
out = append(out, DebtCandidate{Item: item, Check: check})
}
}
return out
}
type DebtCandidate struct {
Item domain.DebtItem `json:"item"`
Check DebtCheck `json:"check"`
}
+68
View File
@@ -0,0 +1,68 @@
package operations
import (
"strings"
"testing"
"orchestra/internal/domain"
)
func debtItem(class domain.DebtClass, kinds ...struct {
kind domain.ObservationKind
task string
}) domain.DebtItem {
item := domain.DebtItem{Class: class}
for _, k := range kinds {
item.Observations = append(item.Observations, domain.DebtObservation{Kind: k.kind, TaskID: k.task})
}
return item
}
type obs = struct {
kind domain.ObservationKind
task string
}
// The thresholds are the whole policy, so each class gets its own case. Polish
// is the one that must never pass, or the ledger becomes a cleanup generator.
func TestDebtEligibilityPerClass(t *testing.T) {
fail := obs{domain.ObservationFailureClass, "t1"}
if !CheckDebtEligibility(debtItem(domain.DebtCorrectness, fail)).Eligible {
t.Fatal("correctness debt is eligible on first observation")
}
twice := debtItem(domain.DebtOperational, fail, obs{domain.ObservationFailureClass, "t1"})
if CheckDebtEligibility(twice).Eligible {
t.Fatal("two occurrences on one task must not qualify as operational debt")
}
spread := debtItem(domain.DebtOperational, fail,
obs{domain.ObservationFailureClass, "t2"}, obs{domain.ObservationFailureClass, "t3"})
if !CheckDebtEligibility(spread).Eligible {
t.Fatal("three occurrences across three tasks must qualify")
}
manual := debtItem(domain.DebtOperational, obs{domain.ObservationManualIntervention, "t1"})
if !CheckDebtEligibility(manual).Eligible {
t.Fatal("one manual intervention must qualify on its own")
}
blocked := debtItem(domain.DebtStructural,
obs{domain.ObservationBlockReason, "t1"}, obs{domain.ObservationBlockReason, "t2"})
if !CheckDebtEligibility(blocked).Eligible {
t.Fatal("two blocked tasks in one component must qualify as structural")
}
polish := debtItem(domain.DebtPolish, fail, fail, fail, fail, fail)
if CheckDebtEligibility(polish).Eligible {
t.Fatal("polish must never promote itself, at any recurrence")
}
}
// A refusal has to say what is missing. "Not eligible" alone sends an operator
// reading code, which is the mistake SubmissionCheck already documents.
func TestDebtRefusalNamesTheMissingEvidence(t *testing.T) {
check := CheckDebtEligibility(debtItem(domain.DebtOperational, obs{domain.ObservationFailureClass, "t1"}))
if check.Eligible || len(check.Reasons) == 0 {
t.Fatalf("want a refusal with reasons, got %+v", check)
}
joined := strings.Join(check.Reasons, " ")
if !strings.Contains(joined, "manual interventions are not recorded") {
t.Fatalf("the refusal must say the intervention count is structurally zero: %v", check.Reasons)
}
}
+227
View File
@@ -0,0 +1,227 @@
package store
import (
"encoding/json"
"sort"
"strings"
"orchestra/internal/domain"
"orchestra/internal/review"
)
// ProjectDebt folds canonical history into a debt ledger. It is a pure
// function of the events so it can be tested against a real log, and it writes
// nothing: this projection is a read model, not a new kind of truth.
//
// It deliberately reports what it cannot see. Worker observations and operator
// interventions are not carried by any event type today, so a ledger that
// stayed silent about them would read as "no operational cost" when it means
// "operational cost is unrecorded".
func ProjectDebt(events []domain.Event) domain.DebtLedger {
return projectDebt(events, nil)
}
// ProjectDebtWithArtifacts is ProjectDebt plus the CAS reads that review
// findings need. Findings are sealed as an artifact rather than inlined in the
// event, and they are the only evidence structural debt has, so a projection
// without them can never produce a structural item at all.
func ProjectDebtWithArtifacts(events []domain.Event, read func(ref string) ([]byte, error)) domain.DebtLedger {
return projectDebt(events, read)
}
func projectDebt(events []domain.Event, readArtifact func(string) ([]byte, error)) domain.DebtLedger {
bySignature := map[string]*domain.DebtItem{}
order := []string{}
seenKinds := map[domain.ObservationKind]bool{}
add := func(class domain.DebtClass, sig string, o domain.DebtObservation, symptom string, sev review.Severity) {
o.Signature = sig
if o.Validate() != nil {
return
}
seenKinds[o.Kind] = true
item, ok := bySignature[sig]
if !ok {
item = &domain.DebtItem{
// The id is the signature, not an Fxx number. This projection
// produces candidates from history; adopting a curated id is a
// later, deliberate step, and minting one here would collide
// with the hand-written ledger.
ID: sig, Class: class, Status: domain.DebtObserved,
Symptom: symptom, Signatures: []string{sig}, Severity: sev,
}
bySignature[sig] = item
order = append(order, sig)
}
item.Observations = append(item.Observations, o)
for _, p := range o.Paths {
if !contains(item.Paths, p) {
item.Paths = append(item.Paths, p)
}
}
}
for _, e := range events {
var p map[string]any
if len(e.Payload) > 0 && json.Unmarshal(e.Payload, &p) != nil {
continue
}
base := domain.DebtObservation{EventID: e.ID, TaskID: e.TaskID, At: e.At}
harness, _ := p["harness_id"].(string)
switch e.Type {
case "TaskBlocked":
reason, _ := p["block_reason"].(string)
if reason == "" {
reason = string(domain.InferBlockReason(str(p["blocker"])))
}
class, ok := domain.DebtClassForBlockReason(domain.BlockReason(reason))
if !ok {
continue
}
o := base
o.Kind = domain.ObservationBlockReason
o.Detail = str(p["blocker"])
add(class, domain.DebtSignature(class, reason, harness, "lease"), o,
"tasks are blocked with "+reason, review.Important)
case "TaskFailed", "TaskReleased":
// A release carries a failure class only when it is a reclaim. A
// handoff release is ordinary progress and never debt.
failure, _ := p["failure_class"].(string)
if failure == "" {
failure, _ = p["reason"].(string)
}
if e.Type == "TaskReleased" && str(p["handoff_ref"]) != "" {
continue
}
class, ok := domain.DebtClassForFailureClass(failure)
if !ok {
continue
}
o := base
o.Kind = domain.ObservationFailureClass
o.Detail = str(p["last_error"])
add(class, domain.DebtSignature(class, failure, harness, "lease"), o,
"tasks end in "+failure, review.Important)
case domain.EventPlanMismatchRecorded:
o := base
o.Kind = domain.ObservationPlanMismatch
o.Detail = str(p["observed"])
o.Paths = strList(p["evidence"])
component := firstComponent(o.Paths)
add(domain.DebtCorrectness,
domain.DebtSignature(domain.DebtCorrectness, "plan_mismatch", "", component), o,
"the repository contradicts sealed plans", review.Important)
case domain.EventReviewRecorded:
// Repeated findings in one component are the only mechanical
// evidence that structure is costing future changes. One finding
// is a review doing its job.
if readArtifact == nil {
break
}
raw, err := readArtifact(str(p["artifact_ref"]))
if err != nil {
break
}
result, err := review.Decode(raw)
if err != nil {
break
}
for _, f := range result.Findings {
component := domain.DebtComponent(f.File)
o := base
o.Kind = domain.ObservationReviewFinding
o.Detail = f.Claim
if f.File != "" {
o.Paths = []string{f.File}
}
add(domain.DebtStructural,
domain.DebtSignature(domain.DebtStructural, string(f.Severity), "", component), o,
"review keeps finding "+string(f.Severity)+" issues in "+component, f.Severity)
}
case domain.EventDeferredFindingRecorded:
o := base
o.Kind = domain.ObservationDeferredFinding
o.Detail = str(p["summary"])
// Polish until something demonstrates a cost. A discovery nobody
// has paid for yet is not debt with a priority.
add(domain.DebtPolish,
domain.DebtSignature(domain.DebtPolish, "deferred_finding", "", ""), o,
"out-of-scope discoveries recorded and unaddressed", review.Minor)
}
}
items := make([]domain.DebtItem, 0, len(order))
for _, sig := range order {
items = append(items, *bySignature[sig])
}
sort.SliceStable(items, func(i, j int) bool {
if items[i].Recurrence() != items[j].Recurrence() {
return items[i].Recurrence() > items[j].Recurrence()
}
return items[i].ID < items[j].ID
})
return domain.DebtLedger{Items: items, Gaps: debtGaps(seenKinds), Events: len(events)}
}
// debtGaps separates two different silences. A kind no event type carries is a
// hole in the system. A kind the log could carry and does not is a fact about
// this history.
func debtGaps(seen map[domain.ObservationKind]bool) []domain.EvidenceGap {
gaps := []domain.EvidenceGap{
{Kind: domain.ObservationManualIntervention, Durable: false,
Reason: "no event type records an operator repair, so every manual recovery is invisible to this ledger"},
{Kind: domain.ObservationWorkerFailure, Durable: false,
Reason: "worker observations live in worker memory and reach the coordinator only inside WorkerHealth, which is not persisted"},
}
for _, k := range []domain.ObservationKind{
domain.ObservationBlockReason, domain.ObservationFailureClass,
domain.ObservationReviewFinding, domain.ObservationPlanMismatch,
domain.ObservationDeferredFinding,
} {
if !seen[k] {
gaps = append(gaps, domain.EvidenceGap{Kind: k, Durable: true,
Reason: "carried by the log, but this history contains none"})
}
}
return gaps
}
// DebtLedger projects the whole log. Read-only, like every other projection
// the coordinator serves.
func (s *Store) DebtLedger() domain.DebtLedger {
return ProjectDebtWithArtifacts(s.Events(0), s.Artifact)
}
func str(v any) string { s, _ := v.(string); return s }
func strList(v any) []string {
list, ok := v.([]any)
if !ok {
return nil
}
out := make([]string, 0, len(list))
for _, item := range list {
if s := strings.TrimSpace(str(item)); s != "" {
out = append(out, s)
}
}
return out
}
func firstComponent(paths []string) string {
for _, p := range paths {
if c := domain.DebtComponent(p); c != "" {
return c
}
}
return ""
}
func contains(list []string, s string) bool {
for _, item := range list {
if item == s {
return true
}
}
return false
}
+87
View File
@@ -0,0 +1,87 @@
package store
import (
"encoding/json"
"testing"
"time"
"orchestra/internal/domain"
)
func debtEvent(id, typ, task string, payload map[string]any) domain.Event {
b, _ := json.Marshal(payload)
return domain.Event{ID: id, Type: typ, TaskID: task, Payload: b, At: time.Now().UTC()}
}
// The opencode shape from run 14: one harness failing the same way across
// several tasks. That is the recurrence the ledger exists to notice.
func TestProjectDebtGroupsOneFailureShapeAcrossTasks(t *testing.T) {
events := []domain.Event{
debtEvent("e1", "TaskFailed", "t1", map[string]any{"failure_class": "retry_limit", "harness_id": "workpc-opencode"}),
debtEvent("e2", "TaskFailed", "t2", map[string]any{"failure_class": "retry_limit", "harness_id": "workpc-opencode"}),
debtEvent("e3", "TaskFailed", "t3", map[string]any{"failure_class": "retry_limit", "harness_id": "workpc-opencode"}),
debtEvent("e4", "TaskFailed", "t4", map[string]any{"failure_class": "retry_limit", "harness_id": "workpc-claude"}),
// A handoff release is ordinary progress and must not become debt.
debtEvent("e5", "TaskReleased", "t5", map[string]any{"handoff_ref": "sha256:abc", "harness_id": "workpc-claude"}),
}
ledger := ProjectDebt(events)
if len(ledger.Items) != 2 {
t.Fatalf("want one item per harness, got %d: %+v", len(ledger.Items), ledger.Items)
}
top := ledger.Items[0]
if top.Recurrence() != 3 || top.AffectedTasks() != 3 {
t.Fatalf("recurrence %d across %d tasks, want 3 and 3", top.Recurrence(), top.AffectedTasks())
}
if top.Class != domain.DebtOperational {
t.Fatalf("class %q, want operational", top.Class)
}
if got := top.ID; got != "v1:operational:retry_limit:workpc-opencode:lease" {
t.Fatalf("signature %q", got)
}
for _, o := range top.Observations {
if o.EventID == "" || o.LegacyRef != "" {
t.Fatalf("observation lost its event provenance: %+v", o)
}
}
}
// A lifecycle stop is the system working. Waiting for a human is not debt, and
// counting it would drown the real signal.
func TestProjectDebtIgnoresOrdinaryLifecycleStops(t *testing.T) {
events := []domain.Event{
debtEvent("e1", "TaskBlocked", "t1", map[string]any{"block_reason": "human_decision", "blocker": "which route"}),
debtEvent("e2", "TaskBlocked", "t2", map[string]any{"block_reason": "trajectory_gate", "blocker": "confirm plan"}),
debtEvent("e3", "TaskBlocked", "t3", map[string]any{"block_reason": "lease_expired", "blocker": "lease expired"}),
}
ledger := ProjectDebt(events)
if len(ledger.Items) != 1 || ledger.Items[0].Class != domain.DebtOperational {
t.Fatalf("want only the lease_expired item, got %+v", ledger.Items)
}
}
// Incompleteness is part of the result. A ledger that stays silent about what
// it cannot record reads as "no operator cost" when it means "operator cost is
// not recorded anywhere".
func TestProjectDebtReportsWhatItCannotSee(t *testing.T) {
ledger := ProjectDebt(nil)
var manual, worker bool
for _, g := range ledger.Gaps {
if g.Durable {
continue
}
switch g.Kind {
case domain.ObservationManualIntervention:
manual = true
case domain.ObservationWorkerFailure:
worker = true
}
}
if !manual || !worker {
t.Fatalf("the two known holes must always be reported: %+v", ledger.Gaps)
}
for _, g := range ledger.Gaps {
if g.Reason == "" {
t.Fatalf("gap %q has no reason", g.Kind)
}
}
}
+7
View File
@@ -424,6 +424,13 @@ func (s *Store) apply(e domain.Event) error {
t.Lease = nil
case "TaskBlocked", "TaskNeedsAttention":
if e.Type == "TaskBlocked" {
if t.Lease != nil {
// Same reason as TaskReleased: a worker may hold a pushed
// anchor whose commit was refused. A reopen returns the task
// to the queue, and the late-handoff path can only accept it
// if the epoch that ended is still on record.
t.LastLeaseEpoch = t.Lease.Epoch
}
t.State = domain.StateBlocked
t.Lease = nil
} else {
+30
View File
@@ -693,3 +693,33 @@ func TestExpiryRetainsLeaseEpoch(t *testing.T) {
t.Fatalf("last lease epoch %q, want %q", after.LastLeaseEpoch, epoch)
}
}
// A worker can hold a pushed anchor whose commit was refused when an operator
// blocks the task. A reopen returns it to the queue, and the late-handoff path
// can only accept that exact owner if the epoch that ended is still recorded.
func TestBlockRetainsLeaseEpochForALaterReopen(t *testing.T) {
s, err := Open(t.TempDir())
if err != nil {
t.Fatal(err)
}
if err := s.Append(created("e1")); err != nil {
t.Fatal(err)
}
id := s.Tasks()[0].ID
if _, err := s.Lease(id, "h1", time.Minute); err != nil {
t.Fatal(err)
}
leased, _ := s.Task(id)
epoch := leased.Lease.Epoch
p, _ := json.Marshal(map[string]any{"blocker": "parked by the operator", "harness_id": "h1", "lease_epoch": epoch})
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskBlocked", TaskID: id, Version: leased.Version + 1, Payload: p, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
after, _ := s.Task(id)
if after.State != domain.StateBlocked || after.Lease != nil {
t.Fatalf("expected a blocked unleased task, got %s lease=%v", after.State, after.Lease)
}
if after.LastLeaseEpoch != epoch || epoch == "" {
t.Fatalf("last lease epoch %q, want %q", after.LastLeaseEpoch, epoch)
}
}