feat(inference): capability-aware routing for tool-heavy stages
Add CapabilityAwareRoutingStrategy — it hard-filters to providers that declare every required capability (like FirstAvailable) but ranks the matches by summed required-capability score; ties keep list order, so it is a strict superset of first-available. The server now wires it as the routing policy. The orchestrator augments a stage's required capabilities with ModelCapability.ToolCalling when the stage grants tools, so tool-heavy stages route to the best tool-calling model rather than whichever healthy provider comes first. Scores come from each provider's declared capabilities(); observed per-model reliability (GET /metrics/tool-reliability) can feed in later as an additional weight. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -11,7 +11,7 @@ Adapter for LLM inference backends. Implements `core:inference` interfaces. No d
|
||||
## Local Contracts
|
||||
|
||||
- `DefaultProviderRegistry` implements `ProviderRegistry` from `core:inference`.
|
||||
- `FirstAvailableRoutingStrategy` is the default routing policy; extend only in `core:inference`, not here.
|
||||
- Two `RoutingStrategy` implementations (both hard-filter to providers declaring every required capability): `FirstAvailableRoutingStrategy` picks the first match; `CapabilityAwareRoutingStrategy` ranks the matches by summed required-capability score (ties keep list order, so it's a superset of first-available). The server wires `CapabilityAwareRoutingStrategy` so tool-heavy stages (which request `ModelCapability.ToolCalling`) route to the best tool-caller. Extend the `RoutingStrategy` interface only in `core:inference`, not here.
|
||||
- All LLM responses are proposals — they must be validated by the core before affecting state (invariant #7). Adapters return raw responses; they do not validate.
|
||||
- Network calls to inference backends are environment observations; results must be recorded as events by callers if replay must reproduce them (invariant #9).
|
||||
- Model lifecycle (spawn/own the llama-server process) lives in `llama_cpp/`; the autonomous scheduler is intentionally unbuilt (see root CLAUDE.md).
|
||||
|
||||
Reference in New Issue
Block a user