feat(inference): capability-aware routing for tool-heavy stages

Add CapabilityAwareRoutingStrategy — it hard-filters to providers that declare every
required capability (like FirstAvailable) but ranks the matches by summed required-capability
score; ties keep list order, so it is a strict superset of first-available. The server now
wires it as the routing policy.

The orchestrator augments a stage's required capabilities with ModelCapability.ToolCalling
when the stage grants tools, so tool-heavy stages route to the best tool-calling model rather
than whichever healthy provider comes first.

Scores come from each provider's declared capabilities(); observed per-model reliability
(GET /metrics/tool-reliability) can feed in later as an additional weight.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-29 11:06:20 +00:00
parent 238d353653
commit 82674e85c4
6 changed files with 122 additions and 4 deletions
@@ -76,7 +76,7 @@ import com.correx.core.events.types.SessionId
import com.correx.infrastructure.InfrastructureModule
import com.correx.infrastructure.inference.DefaultProviderRegistry
import com.correx.infrastructure.workflow.ExecutionPlanCompiler
import com.correx.infrastructure.inference.FirstAvailableRoutingStrategy
import com.correx.infrastructure.inference.CapabilityAwareRoutingStrategy
import com.correx.infrastructure.inference.commons.ManagedInferenceRouter
import com.correx.infrastructure.inference.commons.AmdResourceProbe
import com.correx.infrastructure.inference.commons.NvidiaResourceProbe
@@ -170,7 +170,7 @@ fun main() {
require(staticProviders.isNotEmpty()) { "At least one provider must be configured" }
firstProvider = staticProviders.first()
infraRegistry = InfrastructureModule.createProviderRegistry(staticProviders)
inferenceRouter = DefaultInferenceRouter(infraRegistry, FirstAvailableRoutingStrategy())
inferenceRouter = DefaultInferenceRouter(infraRegistry, CapabilityAwareRoutingStrategy())
llamaBaseUrl = correxConfig.providers.firstOrNull()?.url
?: System.getenv("CORREX_LLAMA_URL")
?: "http://127.0.0.1:10000"