feat(inference): capability-aware routing for tool-heavy stages
Add CapabilityAwareRoutingStrategy — it hard-filters to providers that declare every required capability (like FirstAvailable) but ranks the matches by summed required-capability score; ties keep list order, so it is a strict superset of first-available. The server now wires it as the routing policy. The orchestrator augments a stage's required capabilities with ModelCapability.ToolCalling when the stage grants tools, so tool-heavy stages route to the best tool-calling model rather than whichever healthy provider comes first. Scores come from each provider's declared capabilities(); observed per-model reliability (GET /metrics/tool-reliability) can feed in later as an additional weight. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+7
-1
@@ -106,6 +106,7 @@ import com.correx.core.inference.ResponseFormat
|
||||
import com.correx.core.inference.Tokenizer
|
||||
import com.correx.core.inference.ToolCallRequest
|
||||
import com.correx.core.inference.ToolDefinition
|
||||
import com.correx.core.inference.ModelCapability
|
||||
import com.correx.core.sessions.ApprovalMode
|
||||
import com.correx.core.inference.ToolFunction
|
||||
import com.correx.core.kernel.execution.WorkflowResult
|
||||
@@ -1753,7 +1754,12 @@ abstract class SessionOrchestrator(
|
||||
responseFormat: ResponseFormat = ResponseFormat.Text,
|
||||
effectives: RunEffectives = RunEffectives(toolRegistry, toolExecutor, workspacePolicy),
|
||||
): InferenceResult {
|
||||
val provider = inferenceRouter.route(stageId, stageConfig.requiredCapabilities, stageConfig.modelId)
|
||||
// A stage that grants tools needs a tool-calling model, so request ToolCalling on top of any
|
||||
// declared capabilities — the capability-aware strategy then ranks eligible providers by their
|
||||
// ToolCalling score and routes the stage to the best tool-caller.
|
||||
val requiredCapabilities = stageConfig.requiredCapabilities +
|
||||
if (stageConfig.allowedTools.isNotEmpty()) setOf(ModelCapability.ToolCalling) else emptySet()
|
||||
val provider = inferenceRouter.route(stageId, requiredCapabilities, stageConfig.modelId)
|
||||
log.debug(
|
||||
"[Orchestrator] inference session={} stage={} provider={} timeoutMs={}",
|
||||
sessionId.value, stageId.value, provider.id.value, timeoutMs,
|
||||
|
||||
Reference in New Issue
Block a user