diff --git a/BURNIN.md b/BURNIN.md index 7fa8eac..c7d372a 100644 --- a/BURNIN.md +++ b/BURNIN.md @@ -2479,7 +2479,8 @@ stops being latency and becomes model compatibility. ### opencode is not running, and the reason was only in the pane Task `06G4M6HH5BAM235AC4PWYXF1HM` failed at attempt 3 having never left -`frame`. Its worktree held only `launch.md`. The pane said why: +`frame`. Its worktree held only `launch.md`. The pane said why. The blocked +thing is the selected model, not the OpenCode Zen free tier, which works: ```text This model is not available in your country. @@ -2499,7 +2500,9 @@ fail. Two things are worth keeping. not Orchestra's job; surfacing the capture is. **Do not route to `workpc-opencode` until its model is reachable.** Every task -sent there burns three attempts and then fails. +sent there burns three attempts and then fails. `~/.config/opencode/opencode.jsonc` +sets no top-level `model`, so OpenCode picks its own default. The file already +declares reachable `nvidia-nim` and `llama-cpp` providers. ### The retry budget counts idleness, not just failure diff --git a/DEBT-DESIGN.md b/DEBT-DESIGN.md new file mode 100644 index 0000000..4475b6b --- /dev/null +++ b/DEBT-DESIGN.md @@ -0,0 +1,301 @@ +# Debt ledger: design from what the tree already supports + +Written 2026-08-29 against `3c7cf95`. Read with `BURNIN.md` (the run ledger), +`PLAN-SPEC-DESIGN.md` and `AUDIT.md`. + +This is a design, not an implementation. Nothing here has been built. + +## The problem being solved + +AI raises implementation throughput faster than it raises integration, cleanup, +observability, consolidation and architectural understanding. The residue is +fixes, compatibility paths, operational workarounds, duplicate config, adapter +gaps and one-off patches. Some are justified. Some become permanent because +nobody remembers why they were added or what they have cost since. + +The target is a first-class debt ledger driven by evidence from real tasks, not +a second TODO list. + +## Classes + +Four, and no more. A fifth class invented at runtime makes the priority model +meaningless. + +| Class | Meaning | +|---|---| +| correctness | Known behaviour is wrong or violates an invariant. | +| operational | The system works, but diagnosis, recovery, deployment, observability or operation repeatedly costs time. | +| structural | Duplication or architecture is demonstrably raising the cost of future changes. | +| polish | Cleanup or consistency work with no demonstrated cost yet. | + +Correctness and operational debt gain priority quickly. Structural debt needs +evidence that it is causing repeated work. Polish never becomes work on its own. + +## 1. What Orchestra already has + +The event log is the durable spine, and it already carries most of what a debt +ledger needs. Every row below is mechanically countable today, with no new +instrumentation. + +| Evidence | Source | Carries | +|---|---|---| +| Failure shape | `TaskFailed`, `TaskBlocked` | 12 typed `block_reason` values, 6 `failure_class` values | +| Retry cost | `Task.Attempt`, `NextRetryAt`, `FailureClass` | how many leases a task burned, durable in the projection | +| Review findings | `ReviewRecorded` | `{ID, Severity, File, Line, Claim, Evidence}`, bound to a `ResultSHA` | +| Out-of-scope discoveries | `DeferredFindingRecorded` | `{Summary, Why}`, already recorded outside agent context | +| Plan contradicted by code | `PlanMismatchRecorded` | `{PlanRef, PhaseID, AtSHA, Observed, Contradicts, Evidence[]}` | +| Verification history | `PlanPhaseVerified` | `{Commands, ExitCodes, AtSHA, EvidenceRef}` per phase | +| Rework rounds | `TaskChangesRequested`, `TaskSubmitted` | how many times a change went back | +| Provenance | `TaskCreated` | `Source`, `ExternalID`, `Parent` | + +Three existing patterns matter more than the data. + +**Global-subject events already work.** `QuotaReported` and `StandupAdvisory` +use `TaskID: "system"` and are whitelisted in `internal/domain/domain.go:279`. +A debt item needs no new subject mechanism. + +**`GenerateStandupAdvisory` is the precedent for the shape.** It is a +persisted, read-only recommendation, and applying it is a separate +approval-gated operation. A debt ledger is that pattern with better inputs. + +**`CheckSubmission` is the precedent for eligibility.** It is a pure function +of task, commit and gate run, returning `{Eligible bool, Reasons []string}`. +Debt eligibility should have the same shape. + +## 2. What evidence is missing + +**Worker observations are not durable.** This is the largest gap. The F18 ring +lives in worker memory and reaches the coordinator inside `WorkerHealth` on +heartbeat. `Registry.persistedState` holds captures, commands and workers only, +with no health. A coordinator restart erases every observation. + +Operational debt is exactly what that ring holds. Repeated 409s, unrenewed +leases, adapter gaps. + +**No operator-intervention record.** Every manual repair in run 12 and run 13 +left no trace in Orchestra. State-file edits, worker restarts, two manual +transaction cleanups. "Required manual recovery" is the strongest priority +signal available, and it is currently unrecorded. + +**No diagnosis-cost signal.** Time spent diagnosing is not measured. The +closest proxy is wall time in `blocked` or `needs_attention`. + +**No component dimension.** Findings carry file paths. Tasks carry none. +Breadth across components has to be derived from paths. + +**No link from a repair commit back to what it repaired.** The Fxx-to-commit +mapping exists only in `BURNIN.md` prose. + +**Deferred findings are too thin.** `{Summary, Why}` has no class, no severity, +no paths and no evidence refs. + +## 3. Minimal durable data model + +No new subsystem. Two event types on the existing spine, one projection beside +`Task`. + +```go +// DebtObservation is one piece of counted evidence, always pointing at the +// event that produced it. Provenance is an event id, never prose. +type DebtObservation struct { + EventID string `json:"event_id"` + TaskID string `json:"task_id,omitempty"` + Kind string `json:"kind"` // block_reason, failure_class, review_finding, + // plan_mismatch, manual_intervention, deferred_finding + Signature string `json:"signature"` // the mechanical dedup key, see section 4 + Detail string `json:"detail"` + Paths []string `json:"paths,omitempty"` + At time.Time `json:"at"` +} + +type DebtItem struct { + ID string `json:"id"` // continues the Fxx namespace + Class DebtClass `json:"class"` // correctness, operational, structural, polish + Status DebtStatus `json:"status"` // observed, eligible, scheduled, repaired, withdrawn + Symptom string `json:"symptom"` + Consequence string `json:"consequence"` + Severity Severity `json:"severity"` // reuse review.Severity + Paths []string `json:"paths,omitempty"` + Signatures []string `json:"signatures"` // every key that attaches here + Observations []DebtObservation `json:"observations"` + IntroducedIn string `json:"introduced_in,omitempty"` // task id or commit + RepairBoundary string `json:"repair_boundary,omitempty"` + RepairTask string `json:"repair_task,omitempty"` + RepairCommit string `json:"repair_commit,omitempty"` +} +``` + +Every counted field the priority model needs is derived, never stored: + +```text +recurrence len(Observations) +blocked_tasks distinct TaskID where Kind is block_reason +manual_interventions count of Kind == manual_intervention +breadth distinct components derived from Paths +``` + +A stored count invites drift from the log. A derived one cannot drift. + +Two events carry it. `DebtObserved` appends one observation. `DebtItemUpdated` +records class, status, severity or a merge. Both need adding to the `allowed` +map and a validator, which is the work every existing event type already did. + +## 4. Deduplication without corrupting provenance + +**A signature is computed from typed facts, never from prose.** + +```text +operational class + block_reason + harness + normalized component +correctness class + failure_class + normalized component +structural class + finding severity + normalized component +``` + +Exact signature match attaches automatically. That is the only automatic path. + +**A model may propose, never merge.** Clustering emits `DebtMergeSuggested` +carrying both item ids and its reasoning. The merge is an operator action or a +policy threshold, recorded as `DebtItemUpdated`. + +**Merges are additive.** The surviving item gains the other's signatures and +observations. The absorbed item becomes `withdrawn` with a pointer, and its +observations keep their original event ids. Nothing is rewritten, so a bad +merge is reversible by reading the log. + +**Every observation names its event.** A debt item's evidence is always +checkable against the log that produced it. That rule is what stops a fuzzy +proposal from becoming an unverifiable claim. + +## 5. Eligibility policy + +A pure function, mirroring `CheckSubmission`. + +```go +func CheckDebtEligibility(item DebtItem) DebtCheck // {Eligible bool, Reasons []string} +``` + +| Class | Becomes eligible when | +|---|---| +| correctness | first confirmed observation, always | +| operational | recurrence >= 3 across >= 2 distinct tasks, or >= 1 manual intervention | +| structural | >= 3 review findings, or >= 2 tasks blocked or reworked in the same component | +| polish | never automatically, operator promotion only | + +`Reasons` lists what fired and what did not. "Not eligible" alone sends an +operator reading code, which is the mistake `SubmissionCheck` already documents. + +Eligible means a task may be created. It does not create one. + +## 6. How maintenance and consolidation fit the existing lifecycle + +A maintenance task is an ordinary task. Source `debt`, external id the debt item +id, phases as usual. + +```text +research → plan → implement → verification → review → submission +``` + +Nothing in the lifecycle changes. One extension is needed: the task's brief must +carry the debt item's evidence. That is the mechanism research and plan +artifacts already use, a CAS ref on the task. + +A consolidation task is the same thing with a research brief that asks: + +- which temporary paths are still necessary? +- which fixes now duplicate each other? +- which compatibility branches are obsolete? +- which config or state is represented twice? +- which abstractions exist only because of defects that have since disappeared? +- what can now be deleted safely? + +Its first output is a deletion plan, never an automatic refactor. Verification +uses the project's existing policy. Review is the normal independent review. + +**The repair gets no discount.** Classification as debt changes what gets +scheduled, never what gets checked. + +### Success metrics for maintenance work + +Lines of code are not a metric. Prefer evidence that can be checked: + +- behaviour preserved +- tests preserved or strengthened +- branches removed +- obsolete types removed +- compatibility code removed +- duplicate config removed +- manual recovery paths eliminated + +## 7. What must not be automated + +- No autonomous cleanup agent, and no repo-wide sweep. +- No LLM score treated as authoritative. Counted facts decide, models propose. +- No automatic merge of debt items. +- No maintenance task from polish without an operator. +- No deletion without an independent review. +- No auto-closing an item because a commit touched the file. Closure needs a + repair task or a stated run of clean evidence. +- No new classes invented at runtime. + +## 8. Migration, so there is one track and not two + +`BURNIN.md` holds 58 Fxx entries in the form `| F52 | commit | prose |`. +`AUDIT.md` holds the older narrative. + +One-time import, by hand, with a class assigned per row. Each becomes a +`DebtItem` with `Status: repaired`, its commit as `RepairCommit`, and one +observation citing the ledger. + +**Keep the namespace.** New items continue at F62. Two numbering schemes would +be the first structural debt the ledger itself creates. + +After import, `BURNIN.md` stops being the item ledger and remains the run +narrative. The defect table becomes generated output from the projection. + +### The history that tests the model + +These are not special cases. They are the check on whether the model can +represent the history this project already has. + +| History | Expected class | Why | +|---|---|---| +| F18, the single `last_error` slot | operational | Repeatedly destroyed causal evidence before it was fixed. | +| F43 to F46, the submission path | correctness | Apparently implemented, actually unreachable. | +| F61, planner learns policy by refusal | operational | Costs a round trip per plan, and becomes model compatibility on weaker planners. | +| Duplicated `quality_gate` config | structural | Only if it keeps causing drift or operator mistakes. | +| Old blocked burn-in tasks | neither | Hygiene, unless one exposes a runtime defect. | +| The blocked release loop, F57 to F60 | correctness and operational | Wrong behaviour, and it required manual cleanup twice. | + +## 9. Smallest slice to live-prove first + +**A read-only projection over the existing log. No new events, no writes, no +schema commitment.** + +```text +internal/operations/debt.go DebtItem, signature, projection over s.Events(0) + CheckDebtEligibility +GET /v1/debt read-only, tui surface +``` + +The proof is that the current log already contains runs 10 to 14. The projection +must independently surface, from history alone: + +- the release loop as operational debt, with its 409 recurrence and two manual + cleanups +- the opencode adapter gap as operational debt, across three failed attempts on + one harness +- the retry-idleness dynamic, attached to the tasks that expired + +A model that cannot reproduce debt already known is wrong, and no schema has +been committed to yet. That is the cheapest place to find out. + +**Slice two** makes worker observations durable, because that is the input the +projection will most visibly lack. + +**Slice three** adds `DebtObserved` and manual-intervention recording. + +## Scope boundary + +This design changes no part of the task lifecycle, phases, leases, review, +submission, federation or the plan machinery. The only extensions it needs are +two event types, one projection, one pure eligibility function and one +read-only endpoint.