feat(inference): capability-aware routing for tool-heavy stages

Add CapabilityAwareRoutingStrategy — it hard-filters to providers that declare every
required capability (like FirstAvailable) but ranks the matches by summed required-capability
score; ties keep list order, so it is a strict superset of first-available. The server now
wires it as the routing policy.

The orchestrator augments a stage's required capabilities with ModelCapability.ToolCalling
when the stage grants tools, so tool-heavy stages route to the best tool-calling model rather
than whichever healthy provider comes first.

Scores come from each provider's declared capabilities(); observed per-model reliability
(GET /metrics/tool-reliability) can feed in later as an additional weight.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-29 11:06:20 +00:00
parent 238d353653
commit 82674e85c4
6 changed files with 122 additions and 4 deletions
@@ -106,6 +106,7 @@ import com.correx.core.inference.ResponseFormat
import com.correx.core.inference.Tokenizer
import com.correx.core.inference.ToolCallRequest
import com.correx.core.inference.ToolDefinition
import com.correx.core.inference.ModelCapability
import com.correx.core.sessions.ApprovalMode
import com.correx.core.inference.ToolFunction
import com.correx.core.kernel.execution.WorkflowResult
@@ -1753,7 +1754,12 @@ abstract class SessionOrchestrator(
responseFormat: ResponseFormat = ResponseFormat.Text,
effectives: RunEffectives = RunEffectives(toolRegistry, toolExecutor, workspacePolicy),
): InferenceResult {
val provider = inferenceRouter.route(stageId, stageConfig.requiredCapabilities, stageConfig.modelId)
// A stage that grants tools needs a tool-calling model, so request ToolCalling on top of any
// declared capabilities — the capability-aware strategy then ranks eligible providers by their
// ToolCalling score and routes the stage to the best tool-caller.
val requiredCapabilities = stageConfig.requiredCapabilities +
if (stageConfig.allowedTools.isNotEmpty()) setOf(ModelCapability.ToolCalling) else emptySet()
val provider = inferenceRouter.route(stageId, requiredCapabilities, stageConfig.modelId)
log.debug(
"[Orchestrator] inference session={} stage={} provider={} timeoutMs={}",
sessionId.value, stageId.value, provider.id.value, timeoutMs,