feat(inference): capability-aware routing for tool-heavy stages
Add CapabilityAwareRoutingStrategy — it hard-filters to providers that declare every required capability (like FirstAvailable) but ranks the matches by summed required-capability score; ties keep list order, so it is a strict superset of first-available. The server now wires it as the routing policy. The orchestrator augments a stage's required capabilities with ModelCapability.ToolCalling when the stage grants tools, so tool-heavy stages route to the best tool-calling model rather than whichever healthy provider comes first. Scores come from each provider's declared capabilities(); observed per-model reliability (GET /metrics/tool-reliability) can feed in later as an additional weight. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -76,7 +76,7 @@ import com.correx.core.events.types.SessionId
|
||||
import com.correx.infrastructure.InfrastructureModule
|
||||
import com.correx.infrastructure.inference.DefaultProviderRegistry
|
||||
import com.correx.infrastructure.workflow.ExecutionPlanCompiler
|
||||
import com.correx.infrastructure.inference.FirstAvailableRoutingStrategy
|
||||
import com.correx.infrastructure.inference.CapabilityAwareRoutingStrategy
|
||||
import com.correx.infrastructure.inference.commons.ManagedInferenceRouter
|
||||
import com.correx.infrastructure.inference.commons.AmdResourceProbe
|
||||
import com.correx.infrastructure.inference.commons.NvidiaResourceProbe
|
||||
@@ -170,7 +170,7 @@ fun main() {
|
||||
require(staticProviders.isNotEmpty()) { "At least one provider must be configured" }
|
||||
firstProvider = staticProviders.first()
|
||||
infraRegistry = InfrastructureModule.createProviderRegistry(staticProviders)
|
||||
inferenceRouter = DefaultInferenceRouter(infraRegistry, FirstAvailableRoutingStrategy())
|
||||
inferenceRouter = DefaultInferenceRouter(infraRegistry, CapabilityAwareRoutingStrategy())
|
||||
llamaBaseUrl = correxConfig.providers.firstOrNull()?.url
|
||||
?: System.getenv("CORREX_LLAMA_URL")
|
||||
?: "http://127.0.0.1:10000"
|
||||
|
||||
Reference in New Issue
Block a user