fix(inference): tolerate a briefly-absent provider instead of killing the session (#299)

A transient provider connection drop (e.g. OOM-killed llama.cpp mid-request)
collapsed into a hard NoEligibleProviderException that escaped runInference's
try/catch (the route() call sat outside it), propagated past the stage retry
loop, and landed in ServerModule's session-level catch — failing the WHOLE
session even though the failure was retryable and the provider recovered
seconds later.

- SessionOrchestrator.runInference: wrap router.route() in try/catch so
  routing failures become InferenceResult.Failed and flow through the normal
  retryable/backoff/exhaustion machinery instead of escaping.
- DefaultInferenceRouter.route: distinguish "capability never configured on
  any provider" (fail fast, waiting can't help) from "configured but
  currently unhealthy" (bounded wait/backoff — default 3 attempts x 2s —
  re-checking health before declaring NoEligibleProvider terminal).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GeyGFXczJb8RUWGBKmkm6G
This commit is contained in:
kami
2026-07-21 01:57:04 +04:00
parent d69cb12ce9
commit c5289420a1
4 changed files with 137 additions and 2 deletions
@@ -300,7 +300,15 @@ abstract class SessionOrchestrator(
// ToolCalling score and routes the stage to the best tool-caller.
val requiredCapabilities = stageConfig.requiredCapabilities +
if (withTools && stageConfig.allowedTools.isNotEmpty()) setOf(ModelCapability.ToolCalling) else emptySet()
val provider = inferenceRouter.route(stageId, requiredCapabilities, stageConfig.modelId)
// Routing itself can fail transiently (provider mid-crash-recovery) — it must be retryable
// like any other inference failure, not escape and kill the whole session (see #299).
val provider = try {
inferenceRouter.route(stageId, requiredCapabilities, stageConfig.modelId)
} catch (e: CancellationException) {
throw e
} catch (e: Exception) {
return InferenceResult.Failed(e.message ?: "routing failed")
}
log.debug(
"[Orchestrator] inference session={} stage={} provider={} timeoutMs={}",
sessionId.value, stageId.value, provider.id.value, timeoutMs,