kami
82674e85c4
Add CapabilityAwareRoutingStrategy — it hard-filters to providers that declare every required capability (like FirstAvailable) but ranks the matches by summed required-capability score; ties keep list order, so it is a strict superset of first-available. The server now wires it as the routing policy. The orchestrator augments a stage's required capabilities with ModelCapability.ToolCalling when the stage grants tools, so tool-heavy stages route to the best tool-calling model rather than whichever healthy provider comes first. Scores come from each provider's declared capabilities(); observed per-model reliability (GET /metrics/tool-reliability) can feed in later as an additional weight. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
38 lines
3.0 KiB
Markdown
38 lines
3.0 KiB
Markdown
# infrastructure/inference/
|
|
|
|
## Purpose
|
|
|
|
Inference adapter layer. The root module provides `DefaultProviderRegistry` (registers `InferenceProvider` instances) and `FirstAvailableRoutingStrategy`. Submodules cover shared HTTP client infrastructure (`commons/`), the llama.cpp server adapter (`llama_cpp/`), and the remote OpenAI-compatible adapter (`openai_compat/`, e.g. NVIDIA NIM).
|
|
|
|
## Ownership
|
|
|
|
Adapter for LLM inference backends. Implements `core:inference` interfaces. No dependency on other `infrastructure:*` modules.
|
|
|
|
## Local Contracts
|
|
|
|
- `DefaultProviderRegistry` implements `ProviderRegistry` from `core:inference`.
|
|
- Two `RoutingStrategy` implementations (both hard-filter to providers declaring every required capability): `FirstAvailableRoutingStrategy` picks the first match; `CapabilityAwareRoutingStrategy` ranks the matches by summed required-capability score (ties keep list order, so it's a superset of first-available). The server wires `CapabilityAwareRoutingStrategy` so tool-heavy stages (which request `ModelCapability.ToolCalling`) route to the best tool-caller. Extend the `RoutingStrategy` interface only in `core:inference`, not here.
|
|
- All LLM responses are proposals — they must be validated by the core before affecting state (invariant #7). Adapters return raw responses; they do not validate.
|
|
- Network calls to inference backends are environment observations; results must be recorded as events by callers if replay must reproduce them (invariant #9).
|
|
- Model lifecycle (spawn/own the llama-server process) lives in `llama_cpp/`; the autonomous scheduler is intentionally unbuilt (see root CLAUDE.md).
|
|
- `openai_compat/` is fully remote (no local process/GPU). It speaks `POST {baseUrl}/chat/completions` with `Authorization: Bearer`; `baseUrl` must include the version segment (e.g. `/v1`). It has no `/tokenize`, so it uses a heuristic tokenizer, and no GBNF — JSON artifacts rely on the core's validate-after-retry. Server dispatch keys `[[providers]] type = "nim" | "openai"`; the key comes from `api_key` or `api_key_env`.
|
|
|
|
## Work Guidance
|
|
|
|
Standard adapter rules apply (see parent `AGENTS.md`). HTTP client code uses Ktor CIO; blocking I/O in `withContext(Dispatchers.IO)`.
|
|
|
|
## Verification
|
|
|
|
```
|
|
./gradlew :infrastructure:inference:test --rerun-tasks
|
|
./gradlew :infrastructure:inference:commons:test --rerun-tasks
|
|
./gradlew :infrastructure:inference:llama_cpp:test --rerun-tasks
|
|
./gradlew :infrastructure:inference:openai_compat:test --rerun-tasks
|
|
```
|
|
|
|
## Child DOX Index
|
|
|
|
- `commons/` — shared `ManagedInferenceProvider`, `ModelManager`, `ResourceProbe` (Nvidia/AMD), `ResidencyMode`; no separate AGENTS.md (sub-leaf, covered by this doc)
|
|
- `llama_cpp/` — `LlamaCppInferenceProvider`, `LlamaProcess` (spawns/owns llama-server), `LlamaCppEmbedder`, `LlamaCppTokenizer`, `GbnfGrammarConverter`; no separate AGENTS.md (sub-leaf, covered by this doc)
|
|
- `openai_compat/` — `OpenAiCompatInferenceProvider` (remote Bearer-auth chat completions for NVIDIA NIM/OpenAI), `HeuristicTokenizer`, `OpenAiApiModels`; no separate AGENTS.md (sub-leaf, covered by this doc)
|