fix(kernel): recovery-internal same-file loop-breaker + repair ledger (#309)

Recovery runs freestyle for 3h+/800 events thrashing one file (22
rewrites, 9 rewrites of a single component) with no second
FailureTicketOpenedEvent ever firing, terminated only by human CANCEL.
Root cause: recovery is architected as ONE continuous ReAct loop
(deliberately generous maxToolRounds — most rounds are
re-investigation before the write), so the existing cumulative
tool-failure breaker (stage_loop_break, #78/#304) never fires: each
file_edit call itself SUCCEEDS, only the downstream lsp_diagnostics
gate keeps failing on the same file. The per-gate progress-aware
fingerprint is also unreliable here: it hashes the whole failure-reason
string, which can look "different" each round from unrelated
diagnostics elsewhere even while one file's defect never clears.

Fix — two consumers sharing one event-derived data source
(RecoveryFileLoopBreak.kt, FileRepairOutcome/fileRepairOutcomes):
correlates each FileWrittenEvent with the next LspDiagnosticsCompletedEvent
for the same path (invariant #9 — no re-observation, pure fold).

- Guard (recoveryFileLoopBreak, wired into DefaultSessionOrchestratorStep's
  enterStage before the normal gate-retry machinery): once a single path has
  been rewritten recoveryFileRewriteLimit times (default 3) without its
  diagnostic ever going clean, escalateRecoveryLoop opens a
  FailureTicketOpenedEvent (gate=recovery_loop_break, escalated=true, same
  machinery every other escalation uses) and fails the workflow terminally
  instead of looping again. Recovery is the last tier — there's nowhere
  further to route to.
- Ledger (buildRetryFeedbackEntry in ContextFeedback.kt): each
  already-written file in the trailing repair-mandate slot is now annotated
  with whether rewriting it actually moved the diagnostic, e.g.
  "SessionsList.tsx — written 3x. TS6133: present before AND after every
  write. Re-writing has not changed the result. Change the fix or report
  unresolvable." vs "MainLayout.tsx — written 2x. cleared after write 2.
  done, leave it." Existing trailing-slot precedence (recoveryTicket >
  retryFeedback > groundingFeedback > rejectionFeedback, #313) is untouched —
  this only changes retryFeedback's rendered content, still EntryRole.USER.

FileRepairOutcome/fileRepairOutcomes/describeFileRepairOutcome and the
recovery guard functions live in a new RecoveryFileLoopBreak.kt purely to
keep ContextFeedback.kt and DefaultSessionOrchestratorRecovery.kt under
detekt's per-file function-count threshold (11) — no behavioral reason
for the split.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgDL1v3GuQ9RZnYR6fDT95
This commit is contained in:
2026-07-26 22:58:16 +04:00
parent 2b13f610e1
commit ee69f9becc
7 changed files with 304 additions and 17 deletions
@@ -9,14 +9,12 @@ import com.correx.core.context.model.ContextEntry
import com.correx.core.context.model.ContextLayer import com.correx.core.context.model.ContextLayer
import com.correx.core.context.model.EntryRole import com.correx.core.context.model.EntryRole
import com.correx.core.events.events.FailureTicketOpenedEvent import com.correx.core.events.events.FailureTicketOpenedEvent
import com.correx.core.events.events.FileWrittenEvent
import com.correx.core.events.events.InitialIntentEvent import com.correx.core.events.events.InitialIntentEvent
import com.correx.core.events.events.PlanGroundingEvaluatedEvent import com.correx.core.events.events.PlanGroundingEvaluatedEvent
import com.correx.core.events.events.PlanGroundingVerdict import com.correx.core.events.events.PlanGroundingVerdict
import com.correx.core.events.events.RefinementIterationEvent import com.correx.core.events.events.RefinementIterationEvent
import com.correx.core.events.events.RetryAttemptedEvent import com.correx.core.events.events.RetryAttemptedEvent
import com.correx.core.events.events.StoredEvent import com.correx.core.events.events.StoredEvent
import com.correx.core.events.events.ToolInvocationRequestedEvent
import com.correx.core.events.types.ArtifactId import com.correx.core.events.types.ArtifactId
import com.correx.core.events.types.ContextEntryId import com.correx.core.events.types.ContextEntryId
import com.correx.core.events.types.StageId import com.correx.core.events.types.StageId
@@ -35,15 +33,7 @@ fun buildRetryFeedbackEntry(events: List<StoredEvent>, stageId: StageId): Contex
val latest = events val latest = events
.mapNotNull { it.payload as? RetryAttemptedEvent } .mapNotNull { it.payload as? RetryAttemptedEvent }
.lastOrNull { it.stageId == stageId } ?: return null .lastOrNull { it.stageId == stageId } ?: return null
val stageInvocations = events.mapNotNull { it.payload as? ToolInvocationRequestedEvent } val outcomes = fileRepairOutcomes(events, stageId)
.filter { it.stageId == stageId }
.map { it.invocationId }
.toSet()
val currentImages = events.mapNotNull { it.payload as? FileWrittenEvent }
.filter { it.invocationId in stageInvocations }
.mapNotNull { ev -> ev.postImageHash?.let { ev.path to it } }
.groupBy({ it.first }, { it.second })
.map { (path, hashes) -> path to hashes.last() }
val content = buildString { val content = buildString {
appendLine("## Retry repair state") appendLine("## Retry repair state")
appendLine( appendLine(
@@ -51,15 +41,15 @@ fun buildRetryFeedbackEntry(events: List<StoredEvent>, stageId: StageId): Contex
"'${stageId.value}', gate '${latest.gate}'. The previous attempt failed:", "'${stageId.value}', gate '${latest.gate}'. The previous attempt failed:",
) )
appendLine(latest.failureReason) appendLine(latest.failureReason)
if (currentImages.isNotEmpty()) { if (outcomes.isNotEmpty()) {
appendLine() appendLine()
appendLine( appendLine(
"Files you have already written this stage (authoritative current images — patch " + "Files you have already written this stage (authoritative current state — patch " +
"these, do NOT re-read to rediscover them):", "these, do NOT re-read to rediscover them). Each is annotated with whether " +
"re-writing it actually moved the diagnostic — a file marked unresolved needs a " +
"DIFFERENT fix, not another identical rewrite:",
) )
// Path only — the raw CAS hash is opaque noise the model can't act on and confuses it into outcomes.forEach { o -> appendLine("- ${describeFileRepairOutcome(o)}") }
// reasoning about hashes; it patches by path via file_read/file_write.
currentImages.forEach { (path, _) -> appendLine("- $path") }
} }
append( append(
"Repair the recorded image and the named failure above first. Do not re-discover " + "Repair the recorded image and the named failure above first. Do not re-discover " +
@@ -79,6 +69,10 @@ fun buildRetryFeedbackEntry(events: List<StoredEvent>, stageId: StageId): Contex
) )
} }
// FileRepairOutcome / fileRepairOutcomes / describeFileRepairOutcome moved to
// RecoveryFileLoopBreak.kt (Vikunja #309) — shared with the recovery-stage guard there, and split out
// to keep both this file and DefaultSessionOrchestratorRecovery.kt under detekt's function-count cap.
/** /**
* Feeds the deterministic plan-grounding findings back into the architect stage when the freestyle * Feeds the deterministic plan-grounding findings back into the architect stage when the freestyle
* driver returned its plan for another attempt (grounding verdict != PASS). The findings are already * driver returned its plan for another attempt (grounding verdict != PASS). The findings are already
@@ -75,6 +75,12 @@ internal const val WORKSPACE_PRECONDITION_GATE = "workspace_precondition"
// (never retried in place) by the step handler before the normal retry path. // (never retried in place) by the step handler before the normal retry path.
internal const val STAGE_LOOP_BREAK_GATE = "stage_loop_break" internal const val STAGE_LOOP_BREAK_GATE = "stage_loop_break"
// Gate id for the same-fingerprint loop-breaker firing INSIDE the recovery stage itself (Vikunja
// #309) — a file recovery keeps rewriting without ever clearing its diagnostic. Distinct from
// STAGE_LOOP_BREAK_GATE (which fires on repeated raw TOOL failures pre-recovery): this one is keyed
// on (path, diagnostic) persistence, since recovery's file_edit calls themselves typically succeed.
internal const val RECOVERY_LOOP_BREAK_GATE = "recovery_loop_break"
internal val GATE_REQUIRED_CAPABILITY: Map<String, String> = mapOf( internal val GATE_REQUIRED_CAPABILITY: Map<String, String> = mapOf(
"execution" to "file_write", "execution" to "file_write",
"contract" to "file_write", "contract" to "file_write",
@@ -280,6 +280,9 @@ internal fun DefaultSessionOrchestrator.findRecoveryStage(graph: WorkflowGraph,
id != failingStageId && (cfg.metadata["role"] == "recovery" || id.value == "recovery") id != failingStageId && (cfg.metadata["role"] == "recovery" || id.value == "recovery")
}?.key }?.key
// isRecoveryStage / recoveryFileLoopBreak / escalateRecoveryLoop moved to RecoveryFileLoopBreak.kt
// (Vikunja #309) — split out to keep this file under detekt's function-count cap.
/** /**
* Route-to-owner resolution: map the failing gate's [evidence] (build/tsc output that names * Route-to-owner resolution: map the failing gate's [evidence] (build/tsc output that names
* files, e.g. "src/App.tsx: TS2322") to the stage that most recently WROTE one of those files, * files, e.g. "src/App.tsx: TS2322") to the stage that most recently WROTE one of those files,
@@ -293,6 +293,15 @@ internal suspend fun DefaultSessionOrchestrator.enterStage(
return StepResult.Continue(ctx.copy(stageCount = ctx.stageCount + 1)) return StepResult.Continue(ctx.copy(stageCount = ctx.stageCount + 1))
} }
val refreshedState = orchestrationRepository.getState(ctx.sessionId) val refreshedState = orchestrationRepository.getState(ctx.sessionId)
// #309: inside the recovery stage itself, a same-file/same-diagnostic recurrence is
// never left to loop — recovery has no further tier to route to, so this checks BEFORE
// any of the normal gate-retry machinery (which the whole-reason-string progress
// fingerprint can be fooled into treating as "still progressing" indefinitely).
if (isRecoveryStage(ctx.graph, stageId)) {
recoveryFileLoopBreak(ctx.sessionId, stageId, tuning.recoveryFileRewriteLimit)?.let { reason ->
return StepResult.Terminal(escalateRecoveryLoop(ctx, stageId, result.gate, reason))
}
}
// A repeated missing-build-prerequisite block is not an ordinary retry: it takes a // A repeated missing-build-prerequisite block is not an ordinary retry: it takes a
// bounded, separately-budgeted precondition-resolution path (design #170) that never // bounded, separately-budgeted precondition-resolution path (design #170) that never
// charges the stage retry counter. FallThrough = defer to the normal recovery routing. // charges the stage retry counter. FallThrough = defer to the normal recovery routing.
@@ -59,4 +59,13 @@ data class OrchestrationTuning(
* (no approver connected) auto-rejects the escalated prompt rather than hanging. * (no approver connected) auto-rejects the escalated prompt rather than hanging.
*/ */
val escalateScopeAfterN: Int = 3, val escalateScopeAfterN: Int = 3,
/**
* Inside the recovery stage (Vikunja #309): how many times the SAME file may be rewritten
* without its diagnostic ever clearing before the same-fingerprint loop-breaker escalates and
* fails the run, instead of continuing an unbounded repair loop. Recovery is a single
* continuous ReAct loop (see maxToolRounds doc), so the cumulative tool-failure breaker never
* fires when every file_edit call itself succeeds — only the downstream diagnostic keeps
* failing — hence this separate, path-keyed guard.
*/
val recoveryFileRewriteLimit: Int = 3,
) )
@@ -0,0 +1,140 @@
package com.correx.core.kernel.orchestration
// Split out of ContextFeedback.kt / DefaultSessionOrchestratorRecovery.kt (Vikunja #309) purely to
// stay under detekt's per-file function-count threshold — this is the shared data source (and its
// one consumer that isn't a ContextEntry builder) for the two consumers described below.
import com.correx.core.events.events.FailureTicketOpenedEvent
import com.correx.core.events.events.FileWrittenEvent
import com.correx.core.events.events.LspDiagnosticsCompletedEvent
import com.correx.core.events.events.StoredEvent
import com.correx.core.events.events.ToolInvocationRequestedEvent
import com.correx.core.events.types.SessionId
import com.correx.core.events.types.StageId
import com.correx.core.kernel.execution.WorkflowResult
import com.correx.core.kernel.retry.FailureFingerprint
import com.correx.core.transitions.graph.WorkflowGraph
/**
* Per-file repair outcome for a stage's own writes this stage-entry (Vikunja #309, shared data
* source for two consumers: the retry-feedback ledger in ContextFeedback.kt, and the recovery
* same-fingerprint guard below). Correlates each [FileWrittenEvent] with the next
* [LspDiagnosticsCompletedEvent] recorded for that path (invariant #9 — event-derived, no
* re-observation) to tell "rewriting this file is converging" from "rewriting this file has changed
* nothing" — the closed/open "tab-keeping" a bare file list can't express.
*/
internal data class FileRepairOutcome(
val path: String,
val writeCount: Int,
/** True when the diagnostic run following the LAST write for this path reported no errors. */
val resolved: Boolean,
/** Diagnostic codes present after EVERY write (only meaningful when [resolved] is false). */
val persistentCodes: Set<String>,
/** 1-based write index at which the diagnostic first went clean (only set when [resolved]). */
val clearedAtWrite: Int?,
)
internal fun fileRepairOutcomes(events: List<StoredEvent>, stageId: StageId): List<FileRepairOutcome> {
val invToStage = events.mapNotNull { it.payload as? ToolInvocationRequestedEvent }
.associate { it.invocationId to it.stageId }
val writes = events
.filter { ev ->
val fw = ev.payload as? FileWrittenEvent
fw != null && fw.postImageHash != null && invToStage[fw.invocationId] == stageId
}
.sortedBy { it.sequence }
if (writes.isEmpty()) return emptyList()
val diagRuns = events
.filter { (it.payload as? LspDiagnosticsCompletedEvent)?.stageId == stageId }
.sortedBy { it.sequence }
val paths = writes.map { (it.payload as FileWrittenEvent).path }.distinct()
return paths.map { path ->
val pathWrites = writes.filter { (it.payload as FileWrittenEvent).path == path }
val codesPerWrite = pathWrites.map { w ->
diagRuns.firstOrNull { it.sequence > w.sequence }
?.let { (it.payload as LspDiagnosticsCompletedEvent).diagnostics }
?.filter { d -> d.path == path && d.severity.equals("error", ignoreCase = true) && !d.isLint }
?.mapNotNull { it.code }
?.toSet()
?: emptySet()
}
val resolved = codesPerWrite.last().isEmpty()
FileRepairOutcome(
path = path,
writeCount = pathWrites.size,
resolved = resolved,
persistentCodes = if (resolved) emptySet() else codesPerWrite.reduce { a, b -> a intersect b },
clearedAtWrite = if (resolved) codesPerWrite.indexOfFirst { it.isEmpty() } + 1 else null,
)
}
}
internal fun describeFileRepairOutcome(o: FileRepairOutcome): String {
val header = "${o.path} — written ${o.writeCount}x."
return when {
o.resolved && o.writeCount > 1 -> "$header cleared after write ${o.clearedAtWrite}. done, leave it."
o.resolved -> "$header done, leave it."
o.persistentCodes.isNotEmpty() -> "$header ${o.persistentCodes.joinToString(", ")}: present before " +
"AND after every write. Re-writing has not changed the result. Change the fix or report unresolvable."
else -> "$header diagnostic still failing after the last write."
}
}
/** True when [stageId] is itself declared as the graph's recovery/arbiter stage. */
internal fun DefaultSessionOrchestrator.isRecoveryStage(graph: WorkflowGraph, stageId: StageId): Boolean =
graph.stages[stageId]?.metadata?.get("role") == "recovery"
/**
* Same-fingerprint loop-breaker INSIDE the recovery stage itself (Vikunja #309). Recovery runs as a
* single continuous ReAct loop with a generous round ceiling (see maxToolRounds doc in
* SessionOrchestrator.kt) — so [repeatedToolFailureLoop]'s cumulative TOOL-failure count never fires
* when every individual file_edit call itself succeeds; only the downstream diagnostic gate keeps
* failing on the same file. Detect that instead, via [fileRepairOutcomes]: a path rewritten [limit]
* times whose diagnostic never cleared is provably stuck, independent of the per-gate progress-aware
* fingerprint (which the whole-reason-string comparison can be fooled by — unrelated diagnostics
* elsewhere in the same run make the overall failure text differ each round even though this one
* file's defect never moved).
*/
internal fun DefaultSessionOrchestrator.recoveryFileLoopBreak(
sessionId: SessionId,
stageId: StageId,
limit: Int,
): String? {
val stuck = fileRepairOutcomes(repositories.eventStore.read(sessionId), stageId)
.firstOrNull { !it.resolved && it.writeCount >= limit }
?: return null
val codes = stuck.persistentCodes.takeIf { it.isNotEmpty() }?.joinToString(", ") ?: "its diagnostic"
return "recovery stage ${stageId.value} rewrote '${stuck.path}' ${stuck.writeCount}x without " +
"clearing $codes — the same fix has not changed the result. A materially different fix is " +
"required; escalating instead of continuing to loop."
}
/**
* Terminal escalation for [recoveryFileLoopBreak]. Recovery is the last-resort stage — there is no
* further tier to route to — but the run still opens a [FailureTicketOpenedEvent] (the same
* machinery every other escalation uses, so the stuck file is durably recorded and visible to the
* operator/dashboards) before failing the workflow terminally, rather than looping again.
*/
internal suspend fun DefaultSessionOrchestrator.escalateRecoveryLoop(
ctx: EnrichedExecutionContext,
stageId: StageId,
gate: String,
reason: String,
): WorkflowResult {
emit(
ctx.sessionId,
FailureTicketOpenedEvent(
sessionId = ctx.sessionId,
stageId = stageId,
gate = RECOVERY_LOOP_BREAK_GATE,
category = ticketCategory(gate),
requiredCapability = "file_write",
routeTo = stageId,
evidence = reason,
routeAttempt = 1,
fingerprint = FailureFingerprint.of(reason),
escalated = true,
),
)
return failWorkflow(ctx.sessionId, stageId, reason, retryExhausted = true)
}
@@ -0,0 +1,126 @@
package com.correx.core.kernel.orchestration
import com.correx.core.approvals.Tier
import com.correx.core.events.events.EventMetadata
import com.correx.core.events.events.EventPayload
import com.correx.core.events.events.FileWrittenEvent
import com.correx.core.events.events.LspDiagnostic
import com.correx.core.events.events.LspDiagnosticsCompletedEvent
import com.correx.core.events.events.StoredEvent
import com.correx.core.events.events.ToolInvocationRequestedEvent
import com.correx.core.events.events.ToolRequest
import com.correx.core.events.types.EventId
import com.correx.core.events.types.SessionId
import com.correx.core.events.types.StageId
import com.correx.core.events.types.ToolInvocationId
import kotlinx.datetime.Instant
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Assertions.assertNotNull
import org.junit.jupiter.api.Assertions.assertNull
import org.junit.jupiter.api.Assertions.assertTrue
import org.junit.jupiter.api.Test
/** #309: same-fingerprint loop-breaker + repair-ledger data source, unit-tested as pure folds. */
class RecoveryFileLoopBreakTest {
private val stage = StageId("recovery")
private val session = SessionId("s1")
private var seq = 0L
@Test
fun `a file rewritten past the limit with a diagnostic that never clears trips the breaker`() {
val events = buildList {
repeat(3) { addAll(writeThenDiagnose("SessionsList.tsx", cleared = false, code = "TS6133")) }
}
val reason = recoveryFileLoopBreakPure(events, stage, limit = 3)
assertNotNull(reason)
assertTrue(reason!!.contains("SessionsList.tsx"), "reason: $reason")
assertTrue(reason.contains("3x"), "reason: $reason")
assertTrue(reason.contains("TS6133"), "reason: $reason")
}
@Test
fun `a file whose diagnostic clears before the limit does not trip the breaker`() {
val events = buildList {
addAll(writeThenDiagnose("MainLayout.tsx", cleared = false, code = "TS1005"))
addAll(writeThenDiagnose("MainLayout.tsx", cleared = true))
}
assertNull(recoveryFileLoopBreakPure(events, stage, limit = 3))
}
@Test
fun `ledger annotates an unresolved file distinctly from a resolved one`() {
val events = buildList {
repeat(3) { addAll(writeThenDiagnose("SessionsList.tsx", cleared = false, code = "TS6133")) }
addAll(writeThenDiagnose("MainLayout.tsx", cleared = false, code = "TS1005"))
addAll(writeThenDiagnose("MainLayout.tsx", cleared = true))
}
val outcomes = fileRepairOutcomes(events, stage).associateBy { it.path }
val stuck = outcomes.getValue("SessionsList.tsx")
assertEquals(false, stuck.resolved)
assertEquals(3, stuck.writeCount)
assertEquals(setOf("TS6133"), stuck.persistentCodes)
assertTrue(
describeFileRepairOutcome(stuck).let {
it.contains("written 3x") && it.contains("TS6133") &&
it.contains("Re-writing has not changed the result")
},
)
val fixed = outcomes.getValue("MainLayout.tsx")
assertEquals(true, fixed.resolved)
assertEquals(2, fixed.writeCount)
assertEquals(2, fixed.clearedAtWrite)
assertTrue(describeFileRepairOutcome(fixed).let { it.contains("cleared after write 2") })
}
// recoveryFileLoopBreak is an extension on DefaultSessionOrchestrator that reads the event store;
// its pure core is fileRepairOutcomes, exercised directly here for the same result without needing
// to stand up an orchestrator instance.
private fun recoveryFileLoopBreakPure(events: List<StoredEvent>, stageId: StageId, limit: Int): String? {
val stuck = fileRepairOutcomes(events, stageId).firstOrNull { !it.resolved && it.writeCount >= limit }
?: return null
val codes = stuck.persistentCodes.takeIf { it.isNotEmpty() }?.joinToString(", ") ?: "its diagnostic"
return "recovery stage ${stageId.value} rewrote '${stuck.path}' ${stuck.writeCount}x without " +
"clearing $codes — the same fix has not changed the result. A materially different fix is " +
"required; escalating instead of continuing to loop."
}
private fun ev(payload: EventPayload) = StoredEvent(
metadata = EventMetadata(
eventId = EventId("e${seq++}"),
sessionId = session,
timestamp = Instant.parse("2026-01-01T00:00:00Z"),
schemaVersion = 1,
causationId = null,
correlationId = null,
),
sequence = seq,
sessionSequence = seq,
payload = payload,
)
private fun writeThenDiagnose(path: String, cleared: Boolean, code: String? = null): List<StoredEvent> {
val inv = ToolInvocationId("inv-${seq}")
val req = ev(
ToolInvocationRequestedEvent(
invocationId = inv, sessionId = session, stageId = stage,
toolName = "file_edit", tier = Tier.T2,
request = ToolRequest(inv, session, stage, "file_edit", mapOf("path" to path)),
),
)
val write = ev(
FileWrittenEvent(
invocationId = inv, sessionId = session, path = path,
postImageHash = "h${seq}", preExisted = true, timestampMs = 0,
),
)
val diagnostics = if (cleared) {
emptyList()
} else {
listOf(LspDiagnostic(path = path, line = 1, character = 1, severity = "error", code = code, message = "m"))
}
val diag = ev(LspDiagnosticsCompletedEvent(session, stage, "tsserver", diagnostics))
return listOf(req, write, diag)
}
}