Files

T

kami 82674e85c4 feat(inference): capability-aware routing for tool-heavy stages

Add CapabilityAwareRoutingStrategy — it hard-filters to providers that declare every
required capability (like FirstAvailable) but ranks the matches by summed required-capability
score; ties keep list order, so it is a strict superset of first-available. The server now
wires it as the routing policy.

The orchestrator augments a stage's required capabilities with ModelCapability.ToolCalling
when the stage grants tools, so tool-heavy stages route to the best tool-calling model rather
than whichever healthy provider comes first.

Scores come from each provider's declared capabilities(); observed per-model reliability
(GET /metrics/tool-reliability) can feed in later as an additional weight.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-29 11:06:20 +00:00

3.0 KiB

Raw Blame History

infrastructure/inference/

Purpose

Inference adapter layer. The root module provides DefaultProviderRegistry (registers InferenceProvider instances) and FirstAvailableRoutingStrategy. Submodules cover shared HTTP client infrastructure (commons/), the llama.cpp server adapter (llama_cpp/), and the remote OpenAI-compatible adapter (openai_compat/, e.g. NVIDIA NIM).

Ownership

Adapter for LLM inference backends. Implements core:inference interfaces. No dependency on other infrastructure:* modules.

Local Contracts

DefaultProviderRegistry implements ProviderRegistry from core:inference.
Two RoutingStrategy implementations (both hard-filter to providers declaring every required capability): FirstAvailableRoutingStrategy picks the first match; CapabilityAwareRoutingStrategy ranks the matches by summed required-capability score (ties keep list order, so it's a superset of first-available). The server wires CapabilityAwareRoutingStrategy so tool-heavy stages (which request ModelCapability.ToolCalling) route to the best tool-caller. Extend the RoutingStrategy interface only in core:inference, not here.
All LLM responses are proposals — they must be validated by the core before affecting state (invariant #7). Adapters return raw responses; they do not validate.
Network calls to inference backends are environment observations; results must be recorded as events by callers if replay must reproduce them (invariant #9).
Model lifecycle (spawn/own the llama-server process) lives in llama_cpp/; the autonomous scheduler is intentionally unbuilt (see root CLAUDE.md).
openai_compat/ is fully remote (no local process/GPU). It speaks POST {baseUrl}/chat/completions with Authorization: Bearer; baseUrl must include the version segment (e.g. /v1). It has no /tokenize, so it uses a heuristic tokenizer, and no GBNF — JSON artifacts rely on the core's validate-after-retry. Server dispatch keys [[providers]] type = "nim" | "openai"; the key comes from api_key or api_key_env.

Work Guidance

Standard adapter rules apply (see parent AGENTS.md). HTTP client code uses Ktor CIO; blocking I/O in withContext(Dispatchers.IO).

Verification

./gradlew :infrastructure:inference:test --rerun-tasks
./gradlew :infrastructure:inference:commons:test --rerun-tasks
./gradlew :infrastructure:inference:llama_cpp:test --rerun-tasks
./gradlew :infrastructure:inference:openai_compat:test --rerun-tasks

Child DOX Index

commons/ — shared ManagedInferenceProvider, ModelManager, ResourceProbe (Nvidia/AMD), ResidencyMode; no separate AGENTS.md (sub-leaf, covered by this doc)
llama_cpp/ — LlamaCppInferenceProvider, LlamaProcess (spawns/owns llama-server), LlamaCppEmbedder, LlamaCppTokenizer, GbnfGrammarConverter; no separate AGENTS.md (sub-leaf, covered by this doc)
openai_compat/ — OpenAiCompatInferenceProvider (remote Bearer-auth chat completions for NVIDIA NIM/OpenAI), HeuristicTokenizer, OpenAiApiModels; no separate AGENTS.md (sub-leaf, covered by this doc)

3.0 KiB Raw Blame History