= core-inference == purpose Abstracts LLM inference behind a provider interface, routes requests to healthy providers based on required capabilities, tracks inference lifecycle (started → completed/failed/timed-out) as event-sourced state, and defines the data types used to communicate with inference providers (requests, responses, tool definitions, generation config). == responsibilities * Defining the `InferenceProvider` contract for LLM inference calls * Routing inference requests to the most suitable healthy provider via `InferenceRouter` * Caching provider health checks with configurable TTL * Tracking inference records and their status transitions * Defining shared inference types (`InferenceRequest`, `InferenceResponse`, `GenerationConfig`, `ToolDefinition`, etc.) * Providing tokenizer abstraction for model-family-specific tokenization * Defining model capabilities (`Coding`, `ToolCalling`, `Reasoning`, etc.) for provider selection == non-responsibilities * Does not implement concrete providers — those live in `:infrastructure:*` * Does not manage context construction — uses `ContextPack` from `:core:context` * Does not handle tool execution — only models tool call declarations for the LLM * Does not emit events for model loading failures — those are synchronous exceptions == key types === InferenceProvider * **kind**: interface * **purpose**: Contract for a single LLM provider. Can infer, health-check, and report capabilities. === ProviderRegistry * **kind**: interface * **purpose**: Manages the set of registered `InferenceProvider`s. Supports capability-based resolution. === InferenceRouter * **kind**: interface * **purpose**: Resolves a provider for a given stage and set of required capabilities. Performs live health checks before selection. === DefaultInferenceRouter * **kind**: class * **purpose**: Uses a `RoutingStrategy` to select from healthy candidates. Caches health results with configurable TTL. Re-checks health post-selection to close TOCTOU window. === RoutingStrategy * **kind**: fun interface * **purpose**: Pure selection function — chooses a provider from candidates given required capabilities. Throws `NoEligibleProviderException` if no candidate qualifies. === InferenceRequest * **kind**: data class * **purpose**: Complete input for an LLM inference call. Includes context pack, generation config, tool definitions, and response format. * **fields**: `requestId`, `sessionId`, `stageId`, `contextPack`, `generationConfig`, `timeout`, `responseFormat`, `tools` === InferenceResponse * **kind**: data class * **purpose**: Output of an LLM inference call. * **fields**: `requestId`, `text`, `finishReason`, `tokensUsed`, `latencyMs`, `toolCalls` === GenerationConfig * **kind**: data class * **purpose**: Parameters for LLM generation — all fields required for deterministic replay. * **fields**: `temperature`, `topP`, `maxTokens`, `stopSequences`, `seed` === FinishReason * **kind**: sealed class * **purpose**: Why inference finished. * **variants**: `Stop`, `Length`, `ToolCall`, `Timeout`, `Cancelled`, `Error(message)` === ToolCallRequest * **kind**: data class * **purpose**: A tool call proposed by the LLM. * **fields**: `function` (ToolCallFunction with name and arguments) === ToolDefinition * **kind**: data class * **purpose**: A tool declaration sent to the LLM as part of an inference request. * **fields**: `function` (ToolFunction with name, description, parameters) === ModelCapability * **kind**: sealed class * **purpose**: Broad capability category for provider selection. * **variants**: `Coding`, `ToolCalling`, `Reasoning`, `Summarization`, `General` === CapabilityScore * **kind**: data class * **purpose**: Scores a provider on a single capability (0.0–1.0). === ProviderHealth * **kind**: sealed class * **purpose**: Health status of a provider. * **variants**: `Healthy`, `Degraded(reason)`, `Unavailable(reason)` === InferenceState * **kind**: data class * **purpose**: Event-sourced projection of all inference records for a session. * **fields**: `records` (list of `InferenceRecord`) === InferenceRecord * **kind**: data class * **purpose**: Immutable record of a single inference call. * **fields**: `requestId`, `sessionId`, `stageId`, `providerId`, `status`, `tokensUsed`, `latencyMs`, `failureReason` === InferenceStatus * **kind**: enum * **variants**: `STARTED`, `COMPLETED`, `FAILED`, `TIMED_OUT` === InferenceReducer / DefaultInferenceReducer * **kind**: interface / class * **purpose**: Transforms `InferenceState` given stored events. Handles `InferenceStartedEvent`, `InferenceCompletedEvent`, `InferenceFailedEvent`, `InferenceTimeoutEvent`. === InferenceProjector * **kind**: class * **purpose**: Wraps `InferenceReducer` as a `Projection`. === InferenceRepository * **kind**: class * **purpose**: Rebuilds `InferenceState` for a session via `EventReplayer`. === Tokenizer * **kind**: interface * **purpose**: Model-family-specific tokenization. Provider-owned — two providers for the same capability may not share a tokenizer. === InferenceCancellationToken * **kind**: interface * **purpose**: Cancellation contract for inference. Honour cancellation cooperatively at checkpoints. === ResponseFormat * **kind**: sealed interface * **purpose**: Expected response format from the provider. * **variants**: `Text`, `Json(schema)` === InferenceTimeout * **kind**: value class * **purpose**: Records a deadline duration for timeout cancellation. === CancellationReason * **kind**: sealed class * **purpose**: Why inference was cancelled. * **variants**: `UserRequested`, `StageTimeout`, `SessionCancelled`, `ProviderEvicted` === ModelLoadException * **kind**: class * **purpose**: Thrown when model loading fails. No event is emitted. == event flow *inbound:* * `InferenceStartedEvent` → adds a new record with status `STARTED` * `InferenceCompletedEvent` → sets status to `COMPLETED`, records tokens and latency * `InferenceFailedEvent` → sets status to `FAILED`, records failure reason * `InferenceTimeoutEvent` → sets status to `TIMED_OUT`, records timeout as latency *outbound:* * None. This module is purely a projection — events are emitted by the provider or harness. == integration points * `:core:events` — `StoredEvent`, inference lifecycle event types, `InferenceRequestId`, `ProviderId`, `SessionId`, `StageId` * `:core:context` — `ContextPack` (included in `InferenceRequest`) * `:core:artifacts` — `JsonSchema` (used in `ResponseFormat.Json`) == invariants * `GenerationConfig` has no implicit defaults — all relevant fields must be explicitly specified for deterministic replay * `RoutingStrategy.select()` is a pure function with no I/O or side effects * `DefaultInferenceRouter` never returns a provider whose health check returns `Unavailable` * Provider IDs must be unique across the registry * `InferenceCancellationToken` direction is one-way: a timeout can cancel the token, but cancelling the token must not retroactively signal a timeout * `ModelLoadException` does not emit an event — model loading failure is synchronous and happens before any event would be written * `FinishReason.Cancelled` and `InferenceStatus.TIMED_OUT` are distinct: cancellation can happen for reasons other than timeout == PlantUML diagram [plantuml, core-inference, "png"] ---- include::../../diagrams/core-inference.puml[] ---- == known issues None. == open questions None.