feat(recovery): failure-ticket routing + review-gate RECOVER + return-to-sender

Retry-agency invariant: a stage may only retry a gate it has the capability
to change. A write-less stage failing a build/contract/static gate (e.g.
freestyle final_verification with allowedTools=[shell]) can never fix it, so
retrying in place is futile until the budget drains (Vikunja #41).

Instead: open a FailureTicketOpenedEvent (category + requiredCapability derived
deterministically from the gate id, no LLM) and route to a recovery stage that
holds the capability, bounded by a small per-stage route budget.

- Slice 2: SalvageDecision.RECOVER (ternary CONTINUE/RECOVER/FAIL) lets the
  review-gate salvage judge hand off to recovery. Shared routeToRecovery()
  unifies the deterministic agency guard and the judge on one destination;
  decideGateExhaustion now returns StepResult?.
- Return-to-sender: recovery's exit is dynamic (recoveryReturnMove) — it goes
  back to the exact ticket-origin stage to re-run its gate, so a write-less
  gate anywhere in the graph is handled without skipping intervening stages.
  The synthesized recovery->terminal edge is now only a no-ticket fallback.
- Freestyle wiring: ExecutionPlanCompiler injectRecovery flag (Main passes
  true) synthesizes a write-capable recovery stage; ticket evidence reaches it
  via buildRecoveryTicketEntry.
- Read-only tools always present: StageConfig.effectiveAllowedTools adds
  file_read/list_dir to any tool-granting stage (fixes the verifier reaching
  for `shell ls -R` to inspect the tree).

Tests: RecoveryRoutingTest (deterministic gate, no static return edge — proves
dynamic return) + GateRetryBudgetExhaustionTest RECOVER case + two compiler
tests. All green; detekt clean.
This commit is contained in:
2026-07-07 12:05:26 +04:00
parent 597a26d551
commit 879672a47d
13 changed files with 672 additions and 23 deletions
@@ -69,9 +69,45 @@ data class RetryAttemptedEvent(
val charged: Boolean = true,
) : EventPayload
/** Outcome of the hybrid-exhaustion salvage judgement for the semantic review gate. */
/**
* Outcome of the hybrid-exhaustion salvage judgement for the semantic review gate.
* CONTINUE resets the gate budget and retries the reviewed stage in place; FAIL is terminal;
* RECOVER hands the failure to the recovery stage (same destination as the deterministic-gate
* agency guard) — safe under invariant #7 because recovery output re-enters verification and so
* cannot corrupt state, it only chooses which bounded remediation to spend budget on.
*/
@Serializable
enum class SalvageDecision { CONTINUE, FAIL }
enum class SalvageDecision { CONTINUE, FAIL, RECOVER }
/**
* A verification gate failed on a stage that lacks the capability to change the failure condition
* (the retry-agency invariant: recovery/ticket model). Rather than retry the same write-less stage
* in place — which is futile by construction (live repro: freestyle `final_verification` compiled
* with `allowedTools=[shell]` re-ran the identical broken `npm run build` until the budget drained,
* never able to touch a file) — the orchestrator opens a failure ticket and routes to a recovery
* stage that DOES hold the capability. [category] and [requiredCapability] are derived
* deterministically from [gate] (no LLM), so this drives routing without violating invariant #7.
* [evidence] is the gate's captured failure output; it is recorded here at ticket-open time and
* replay reads it back (invariant #9) — the gate command is never re-run on replay.
*/
@Serializable
@SerialName("FailureTicketOpened")
data class FailureTicketOpenedEvent(
val sessionId: SessionId,
// The stage that failed the gate but lacks the capability to fix it (the write-less verifier).
val stageId: StageId,
val gate: String,
// Deterministic from [gate]: "implementation" | "planning" | "environment".
val category: String,
// The capability the failing stage lacks (e.g. "file_write"); the recovery stage must hold it.
val requiredCapability: String,
// The recovery stage the ticket is routed to.
val routeTo: StageId,
// The gate's captured failure output (command/exit/stderr/findings) that recovery must resolve.
val evidence: String,
// 1-based count of times this stage has been routed to recovery (own, small route budget).
val routeAttempt: Int,
) : EventPayload
/**
* Recorded when a gate exhausts its retry budget and (per design decision 3) the salvage judge is
@@ -25,4 +25,10 @@ data class OrchestrationState(
// Gates that have already spent their one hybrid-exhaustion salvage reset (review gate only,
// see RetrySalvageDecidedEvent) — a second exhaustion for that gate is terminal.
val gateSalvageUsed: Set<String> = emptySet(),
// Recovery-routing budget (retry-agency / ticket model): how many times each stage id has been
// routed to a recovery stage because it failed a gate it lacked the capability to fix (see
// FailureTicketOpenedEvent). Deliberately NOT reset by TransitionExecutedEvent — the
// verification→recovery→verification loop must stay bounded across those transitions, exactly as
// refinementIterations does for back-edge loops.
val recoveryRoutes: Map<String, Int> = emptyMap(),
)
@@ -59,6 +59,7 @@ import com.correx.core.events.events.RepoKnowledgeRetrievedEvent
import com.correx.core.events.events.RepoMapComputedEvent
import com.correx.core.events.events.RetryAttemptedEvent
import com.correx.core.events.events.RetrySalvageDecidedEvent
import com.correx.core.events.events.FailureTicketOpenedEvent
import com.correx.core.events.events.WorkspaceStateObservedEvent
import com.correx.core.events.events.RiskAssessedEvent
import com.correx.core.events.events.SourceFetchedEvent
@@ -146,6 +147,7 @@ val eventModule = SerializersModule {
subclass(WorkflowCompletedEvent::class)
subclass(RetryAttemptedEvent::class)
subclass(RetrySalvageDecidedEvent::class)
subclass(FailureTicketOpenedEvent::class)
subclass(RefinementIterationEvent::class)
subclass(RepoMapComputedEvent::class)
subclass(WorkspaceStateObservedEvent::class)
@@ -8,6 +8,7 @@ import com.correx.core.events.events.RepoKnowledgeHit
import com.correx.core.context.model.ContextEntry
import com.correx.core.context.model.ContextLayer
import com.correx.core.context.model.EntryRole
import com.correx.core.events.events.FailureTicketOpenedEvent
import com.correx.core.events.events.RefinementIterationEvent
import com.correx.core.events.events.RetryAttemptedEvent
import com.correx.core.events.events.StoredEvent
@@ -38,6 +39,33 @@ fun buildRetryFeedbackEntry(events: List<StoredEvent>, stageId: StageId): Contex
)
}
/**
* Feeds the open failure ticket into the recovery stage's context. The recovery stage was entered
* by kernel routing (not a normal edge) precisely because an upstream write-less stage failed a gate
* it lacked the capability to fix; without the ticket the recovery stage has no idea what to repair.
* Surfaces the failing stage, the gate, and the captured gate output (evidence, recorded at
* ticket-open per invariant #9) so recovery can act on recorded fact, not a re-observation.
*/
fun buildRecoveryTicketEntry(events: List<StoredEvent>, stageId: StageId): ContextEntry? {
val ticket = events
.mapNotNull { it.payload as? FailureTicketOpenedEvent }
.lastOrNull { it.routeTo == stageId } ?: return null
val content = "## Recovery ticket\n" +
"Stage '${ticket.stageId.value}' failed the '${ticket.gate}' gate (${ticket.category}) and " +
"lacks the '${ticket.requiredCapability}' capability to fix it, so the failure was routed to " +
"you. Repair the underlying cause, then control returns to verification.\n" +
"### Gate output\n${ticket.evidence}"
return ContextEntry(
id = ContextEntryId(UUID.randomUUID().toString()),
layer = ContextLayer.L1,
content = content,
sourceType = "recoveryTicket",
sourceId = stageId.value,
tokenEstimate = content.length / 4,
role = EntryRole.SYSTEM,
)
}
fun criticArtifactIds(
events: List<StoredEvent>,
graph: WorkflowGraph,
@@ -7,6 +7,7 @@ import com.correx.core.events.events.RefinementIterationEvent
import com.correx.core.events.events.RetryAttemptedEvent
import com.correx.core.events.events.RetrySalvageDecidedEvent
import com.correx.core.events.events.SalvageDecision
import com.correx.core.events.events.FailureTicketOpenedEvent
import com.correx.core.events.events.StoredEvent
import com.correx.core.events.events.TransitionExecutedEvent
import com.correx.core.events.events.WorkflowCompletedEvent
@@ -94,6 +95,12 @@ class DefaultOrchestrationReducer : OrchestrationReducer {
refinementIterations = state.refinementIterations + (p.cycleKey to p.iteration),
)
// Charge the failing stage's recovery-route budget. routeAttempt is the 1-based count the
// orchestrator computed, so fold it directly (idempotent under replay).
is FailureTicketOpenedEvent -> state.copy(
recoveryRoutes = state.recoveryRoutes + (p.stageId.value to p.routeAttempt),
)
else -> state
}
}
@@ -15,6 +15,7 @@ import com.correx.core.events.events.ArtifactContentStoredEvent
import com.correx.core.events.events.ClarificationAnswer
import com.correx.core.events.events.ClarificationAnsweredEvent
import com.correx.core.events.events.ArtifactValidatedEvent
import com.correx.core.events.events.FailureTicketOpenedEvent
import com.correx.core.events.events.OrchestrationResumedEvent
import com.correx.core.events.events.RefinementIterationEvent
import com.correx.core.events.events.RetrySalvageDecidedEvent
@@ -29,6 +30,7 @@ import com.correx.core.events.types.ClarificationRequestId
import com.correx.core.events.types.ArtifactLifecyclePhase
import com.correx.core.events.types.SessionId
import com.correx.core.events.types.StageId
import com.correx.core.events.types.TransitionId
import com.correx.core.kernel.execution.WorkflowResult
import com.correx.core.kernel.orchestration.subagent.InSessionSubagentRunner
import com.correx.core.kernel.orchestration.subagent.SubagentRunRequest
@@ -54,6 +56,25 @@ private val log = LoggerFactory.getLogger(DefaultSessionOrchestrator::class.java
// Fallback refinement-loop cap when a stage does not declare maxRetries.
private const val DEFAULT_MAX_REFINEMENT = 3
// Own, small budget for the retry-agency recovery route: how many times a single stage may be
// routed to a recovery stage before the run fails terminally. Deliberately smaller than the
// per-gate retry budget — recovery is a purpose-built repair-then-reverify loop, not open-ended.
private const val RECOVERY_ROUTE_BUDGET = 2
// Deterministic gate id → capability the failing stage must hold to change the failure condition.
// A gate not listed here is not eligible for recovery routing (retries in place as before).
private val GATE_REQUIRED_CAPABILITY: Map<String, String> = mapOf(
"execution" to "file_write",
"contract" to "file_write",
"static_analysis" to "file_write",
)
// Deterministic failure category from the gate id (no LLM — keeps routing off the untrusted path).
private fun ticketCategory(gate: String): String = when (gate) {
"plan_compile" -> "planning"
else -> "implementation"
}
class DefaultSessionOrchestrator(
private val repositories: OrchestratorRepositories,
engines: OrchestratorEngines,
@@ -138,6 +159,12 @@ class DefaultSessionOrchestrator(
val resolved = resolveTransition(
enriched.graph, enriched.sessionId, enriched.currentStageId, stageArtifacts, artifactContent,
)
// Recovery return-to-sender: a recovery stage is entered by kernel routing from whichever
// write-less stage failed a gate, so its exit must go BACK to that exact stage (re-run its
// gate) — not follow a static edge that would skip intervening stages. Derived from the latest
// failure ticket routed here (replay-deterministic). Overrides the resolved edge; an operator
// preempt below still wins.
val baseDecision = recoveryReturnMove(enriched) ?: resolved
// Preemptive redirect override (freestyle graph re-routing): if the operator confirmed a
// jump (a recorded, unconsumed PreemptRedirectEvent), it overrides the resolver's edge —
// unless the target is unknown or its needs aren't satisfied, in which case it is blocked
@@ -161,9 +188,9 @@ class DefaultSessionOrchestrator(
"[Orchestrator] redirect blocked session={} to={} reason={}",
enriched.sessionId.value, outcome.event.toStageId.value, outcome.event.reason,
)
resolved
baseDecision
}
PreemptRedirect.Outcome.None -> resolved
PreemptRedirect.Outcome.None -> baseDecision
}
log.debug(
"[Orchestrator] transition session={} stage={} decision={}",
@@ -229,6 +256,114 @@ class DefaultSessionOrchestrator(
}
/**
* Retry-agency guard (ticket + recovery model). When [failure]'s gate requires a capability the
* failing stage does not hold (e.g. a build/contract gate needs `file_write` on a write-less
* verifier), retrying the stage in place can never change the outcome. Instead open a
* [FailureTicketOpenedEvent] and route to the graph's recovery stage — which does hold the
* capability — bounded by the stage's own [RECOVERY_ROUTE_BUDGET].
*
* Returns a [StepResult] when it took over the failure (routed, or budget-exhausted terminal),
* or null to fall through to the normal per-gate retry path. Null in every backward-compatible
* case: gate not capability-gated, stage already has the capability, or no recovery stage exists.
*/
private suspend fun maybeRouteToRecovery(
ctx: EnrichedExecutionContext,
stageId: StageId,
failure: StageExecutionResult.Failure,
state: OrchestrationState,
): StepResult? {
val requiredCapability = GATE_REQUIRED_CAPABILITY[failure.gate] ?: return null
val stageTools = ctx.graph.stages[stageId]?.allowedTools ?: emptySet()
if (requiredCapability in stageTools) return null // stage has agency — retry in place is valid
return routeToRecovery(ctx, stageId, failure.gate, requiredCapability, failure.reason, state)
}
/**
* Open a [FailureTicketOpenedEvent] and route [stageId]'s failure to the graph's recovery stage,
* bounded by [RECOVERY_ROUTE_BUDGET]. Shared by the deterministic agency guard
* ([maybeRouteToRecovery]) and the review-gate salvage judge ([decideGateExhaustion] on
* [SalvageDecision.RECOVER]) so both routing paths reach the same recovery destination.
*
* Returns null when the graph declares no recovery stage (caller falls back to its legacy path);
* a terminal [StepResult] when the route budget is spent; otherwise the recovery stage's result.
*/
private suspend fun routeToRecovery(
ctx: EnrichedExecutionContext,
stageId: StageId,
gate: String,
requiredCapability: String,
reason: String,
state: OrchestrationState,
): StepResult? {
val recoveryId = findRecoveryStage(ctx.graph, stageId) ?: return null // no recovery stage: legacy path
val used = state.recoveryRoutes[stageId.value] ?: 0
if (used >= RECOVERY_ROUTE_BUDGET) {
return StepResult.Terminal(
failWorkflow(
ctx.sessionId,
stageId,
"recovery route budget exhausted for stage ${stageId.value} (gate=$gate): $reason",
retryExhausted = true,
),
)
}
emit(
ctx.sessionId,
FailureTicketOpenedEvent(
sessionId = ctx.sessionId,
stageId = stageId,
gate = gate,
category = ticketCategory(gate),
requiredCapability = requiredCapability,
routeTo = recoveryId,
evidence = reason,
routeAttempt = used + 1,
),
)
log.info(
"[Orchestrator] failure ticket session={} stage={} gate={} lacks={} -> route to recovery={} ({}/{})",
ctx.sessionId.value, stageId.value, gate, requiredCapability,
recoveryId.value, used + 1, RECOVERY_ROUTE_BUDGET,
)
val advancedTo = advanceStage(
ctx.sessionId,
stageId,
TransitionDecision.Move(transitionId = TransitionId("recovery-route"), to = recoveryId),
)
return enterStage(ctx.copy(currentStageId = advancedTo), recoveryId)
}
/**
* The graph's recovery stage: a stage marked `metadata["role"] == "recovery"` (or, as a
* convenience, one whose id is literally "recovery"), other than [failingStageId] itself.
* Null when the graph declares none — recovery routing is opt-in per workflow.
*/
private fun findRecoveryStage(graph: WorkflowGraph, failingStageId: StageId): StageId? =
graph.stages.entries.firstOrNull { (id, cfg) ->
id != failingStageId && (cfg.metadata["role"] == "recovery" || id.value == "recovery")
}?.key
/**
* When the current stage is a recovery stage that has finished, route it back to the stage the
* failure came from — the origin of the latest [FailureTicketOpenedEvent] routed here — so the
* gate that triggered the route is re-run. This makes recovery correct for arbitrary graph
* topology (a write-less gate anywhere routes here and returns to exactly its own stage) rather
* than relying on a static recovery→terminal edge that would skip intervening stages. Returns
* null when the current stage is not a recovery stage or no ticket routed here (then the
* resolver's normal edge — the compiler's fallback recovery→terminal — is used).
*/
private fun recoveryReturnMove(ctx: EnrichedExecutionContext): TransitionDecision.Move? {
val cfg = ctx.graph.stages[ctx.currentStageId] ?: return null
val isRecovery = cfg.metadata["role"] == "recovery" || ctx.currentStageId.value == "recovery"
if (!isRecovery) return null
val origin = repositories.eventStore.read(ctx.sessionId)
.mapNotNull { it.payload as? FailureTicketOpenedEvent }
.lastOrNull { it.routeTo == ctx.currentStageId }
?.stageId ?: return null
return TransitionDecision.Move(transitionId = TransitionId("recovery-return"), to = origin)
}
/** Returns the cached validated artifact content for the given session + artifact id, or null if absent. */
fun validatedArtifactContent(sessionId: SessionId, artifactId: ArtifactId): String? =
artifactContentCache["${sessionId.value}:${artifactId.value}"]
@@ -450,6 +585,10 @@ class DefaultSessionOrchestrator(
return StepResult.Continue(ctx.copy(stageCount = ctx.stageCount + 1))
}
val refreshedState = orchestrationRepository.getState(ctx.sessionId)
// Retry-agency invariant: a gate this stage lacks the capability to fix must not
// be retried in place (futile). Route to a recovery stage that holds the
// capability, if one exists and the route budget remains.
maybeRouteToRecovery(ctx, stageId, result, refreshedState)?.let { return it }
val gateDecision = retryCoordinator.decide(
sessionId = ctx.sessionId,
stageId = stageId,
@@ -461,10 +600,9 @@ class DefaultSessionOrchestrator(
when (gateDecision) {
RetryDecision.Retry -> Unit // retry — loop and re-execute
RetryDecision.Exhausted -> {
val terminal = decideGateExhaustion(
ctx.sessionId, stageId, result.gate, result.reason, refreshedState,
)
if (terminal != null) return StepResult.Terminal(terminal)
decideGateExhaustion(
ctx, stageId, result.gate, result.reason, refreshedState,
)?.let { return it }
// review-gate salvage said CONTINUE — budget was reset by the reducer;
// loop and re-execute the stage with a fresh budget.
}
@@ -482,17 +620,19 @@ class DefaultSessionOrchestrator(
* FAIL is terminal. A gate that already spent its one salvage reset (state.gateSalvageUsed) fails
* immediately on a second exhaustion, regardless of what the judge would say.
*
* Returns the terminal [WorkflowResult] to fail with, or null to retry (loop and re-execute).
* Returns a terminal [StepResult] to fail/route with, the recovery stage's result on RECOVER,
* or null to retry in place (loop and re-execute).
*/
private suspend fun decideGateExhaustion(
sessionId: SessionId,
ctx: EnrichedExecutionContext,
stageId: StageId,
gate: String,
reason: String,
state: OrchestrationState,
): WorkflowResult? {
): StepResult? {
val sessionId = ctx.sessionId
if (gate != "review" || state.gateSalvageUsed.contains(gate)) {
return failWorkflow(sessionId, stageId, reason, retryExhausted = true)
return StepResult.Terminal(failWorkflow(sessionId, stageId, reason, retryExhausted = true))
}
// No judge wired: degrade safely with a deterministic allow-one-reset-then-fail policy —
// the gateSalvageUsed check above already ensures this fires at most once per gate.
@@ -504,7 +644,12 @@ class DefaultSessionOrchestrator(
emit(sessionId, RetrySalvageDecidedEvent(sessionId, stageId, gate, judgment.decision, judgment.rationale))
return when (judgment.decision) {
SalvageDecision.CONTINUE -> null
SalvageDecision.FAIL -> failWorkflow(sessionId, stageId, reason, retryExhausted = true)
SalvageDecision.FAIL -> StepResult.Terminal(failWorkflow(sessionId, stageId, reason, retryExhausted = true))
// The judge chose recovery: route to the recovery stage (file_write is the capability it
// provides). Degrade to terminal if the graph declares no recovery stage.
SalvageDecision.RECOVER ->
routeToRecovery(ctx, stageId, gate, "file_write", reason, state)
?: StepResult.Terminal(failWorkflow(sessionId, stageId, reason, retryExhausted = true))
}
}
@@ -505,6 +505,8 @@ abstract class SessionOrchestrator(
val agentInstructionsEntries = emptyList<ContextEntry>()
val retryFeedbackEntries = buildRetryFeedbackEntry(sessionEvents, stageId)
?.let { listOf(it) } ?: emptyList()
val recoveryTicketEntries = buildRecoveryTicketEntry(sessionEvents, stageId)
?.let { listOf(it) } ?: emptyList()
val vocabularyEntries = artifactKindRegistry
?.takeIf { stageConfig.metadata["injectArtifactKinds"] == "true" }
?.let { listOf(buildArtifactKindVocabularyEntry(it.list())) } ?: emptyList()
@@ -532,7 +534,7 @@ abstract class SessionOrchestrator(
systemPrompt + profileEntries + projectProfileEntries + agentInstructionsEntries +
journalEntries + repoMapEntries + claimedTaskEntries +
needsEntries + schemaEntries + vocabularyEntries + promptEntries + steeringEntries +
clarificationEntries + retryFeedbackEntries,
clarificationEntries + retryFeedbackEntries + recoveryTicketEntries,
)
val contextPack = runCatching {
contextPackBuilder.build(
@@ -1021,7 +1023,7 @@ abstract class SessionOrchestrator(
),
)
val deniedByStage = toolCall.function.name != STAGE_COMPLETE_TOOL &&
toolCall.function.name !in stageConfig.allowedTools
toolCall.function.name !in stageConfig.effectiveAllowedTools
val deniedByReadOnly = !deniedByStage && isReadOnlyMode(sessionId) &&
(tool?.requiredCapabilities?.contains(ToolCapability.FILE_WRITE) == true)
if (deniedByStage || deniedByReadOnly) {
@@ -2563,7 +2565,7 @@ abstract class SessionOrchestrator(
tools = if (!withTools) {
emptyList()
} else {
stageConfig.allowedTools
stageConfig.effectiveAllowedTools
.mapNotNull { effectives.registry?.resolve(it) }
.filter { tool ->
// ponytail: filter write tools while read-before-write block is active; restored once a read completes
@@ -52,4 +52,20 @@ data class StageConfig(
// inference), so a docs-only plan is left untouched. Explicit build_expectation always wins.
val autoBuildGate: Boolean = false,
val metadata: Map<String, String> = emptyMap(),
)
) {
/**
* The tools actually permitted at run time: the stage's declared [allowedTools] plus the
* always-available read-only tools ([ALWAYS_AVAILABLE_READ_TOOLS]). A stage that grants any tool
* can always inspect the workspace (read a file, list a dir) without the planner having to
* remember to list them — this is why a write-less verifier could `shell ls -R` (flooding its
* own context) instead of a scoped `list_dir`. A stage that declares NO tools stays toolless (a
* pure-reasoning stage is left as designed — read tools would flip it into tool-calling mode).
*/
val effectiveAllowedTools: Set<String>
get() = if (allowedTools.isEmpty()) allowedTools else allowedTools + ALWAYS_AVAILABLE_READ_TOOLS
companion object {
/** Read-only tools every tool-granting stage may call regardless of its declared set. */
val ALWAYS_AVAILABLE_READ_TOOLS: Set<String> = setOf("file_read", "list_dir")
}
}