= core-inference

== purpose

Abstracts LLM inference behind a provider interface, routes requests to healthy providers based on required capabilities, tracks inference lifecycle (started → completed/failed/timed-out) as event-sourced state, and defines the data types used to communicate with inference providers (requests, responses, tool definitions, generation config).

== responsibilities

* Defining the `InferenceProvider` contract for LLM inference calls
* Routing inference requests to the most suitable healthy provider via `InferenceRouter`
* Caching provider health checks with configurable TTL
* Tracking inference records and their status transitions
* Defining shared inference types (`InferenceRequest`, `InferenceResponse`, `GenerationConfig`, `ToolDefinition`, etc.)
* Providing tokenizer abstraction for model-family-specific tokenization
* Defining model capabilities (`Coding`, `ToolCalling`, `Reasoning`, etc.) for provider selection

== non-responsibilities

* Does not implement concrete providers — those live in `:infrastructure:*`
* Does not manage context construction — uses `ContextPack` from `:core:context`
* Does not handle tool execution — only models tool call declarations for the LLM
* Does not emit events for model loading failures — those are synchronous exceptions

== key types

=== InferenceProvider
* **kind**: interface
* **purpose**: Contract for a single LLM provider. Can infer, health-check, and report capabilities.

=== ProviderRegistry
* **kind**: interface
* **purpose**: Manages the set of registered `InferenceProvider`s. Supports capability-based resolution.

=== InferenceRouter
* **kind**: interface
* **purpose**: Resolves a provider for a given stage and set of required capabilities. Performs live health checks before selection.

=== DefaultInferenceRouter
* **kind**: class
* **purpose**: Uses a `RoutingStrategy` to select from healthy candidates. Caches health results with configurable TTL. Re-checks health post-selection to close TOCTOU window.

=== RoutingStrategy
* **kind**: fun interface
* **purpose**: Pure selection function — chooses a provider from candidates given required capabilities. Throws `NoEligibleProviderException` if no candidate qualifies.

=== InferenceRequest
* **kind**: data class
* **purpose**: Complete input for an LLM inference call. Includes context pack, generation config, tool definitions, and response format.
* **fields**: `requestId`, `sessionId`, `stageId`, `contextPack`, `generationConfig`, `timeout`, `responseFormat`, `tools`

=== InferenceResponse
* **kind**: data class
* **purpose**: Output of an LLM inference call.
* **fields**: `requestId`, `text`, `finishReason`, `tokensUsed`, `latencyMs`, `toolCalls`

=== GenerationConfig
* **kind**: data class
* **purpose**: Parameters for LLM generation — all fields required for deterministic replay.
* **fields**: `temperature`, `topP`, `maxTokens`, `stopSequences`, `seed`

=== FinishReason
* **kind**: sealed class
* **purpose**: Why inference finished.
* **variants**: `Stop`, `Length`, `ToolCall`, `Timeout`, `Cancelled`, `Error(message)`

=== ToolCallRequest
* **kind**: data class
* **purpose**: A tool call proposed by the LLM.
* **fields**: `function` (ToolCallFunction with name and arguments)

=== ToolDefinition
* **kind**: data class
* **purpose**: A tool declaration sent to the LLM as part of an inference request.
* **fields**: `function` (ToolFunction with name, description, parameters)

=== ModelCapability
* **kind**: sealed class
* **purpose**: Broad capability category for provider selection.
* **variants**: `Coding`, `ToolCalling`, `Reasoning`, `Summarization`, `General`

=== CapabilityScore
* **kind**: data class
* **purpose**: Scores a provider on a single capability (0.0–1.0).

=== ProviderHealth
* **kind**: sealed class
* **purpose**: Health status of a provider.
* **variants**: `Healthy`, `Degraded(reason)`, `Unavailable(reason)`

=== InferenceState
* **kind**: data class
* **purpose**: Event-sourced projection of all inference records for a session.
* **fields**: `records` (list of `InferenceRecord`)

=== InferenceRecord
* **kind**: data class
* **purpose**: Immutable record of a single inference call.
* **fields**: `requestId`, `sessionId`, `stageId`, `providerId`, `status`, `tokensUsed`, `latencyMs`, `failureReason`

=== InferenceStatus
* **kind**: enum
* **variants**: `STARTED`, `COMPLETED`, `FAILED`, `TIMED_OUT`

=== InferenceReducer / DefaultInferenceReducer
* **kind**: interface / class
* **purpose**: Transforms `InferenceState` given stored events. Handles `InferenceStartedEvent`, `InferenceCompletedEvent`, `InferenceFailedEvent`, `InferenceTimeoutEvent`.

=== InferenceProjector
* **kind**: class
* **purpose**: Wraps `InferenceReducer` as a `Projection<InferenceState>`.

=== InferenceRepository
* **kind**: class
* **purpose**: Rebuilds `InferenceState` for a session via `EventReplayer`.

=== Tokenizer
* **kind**: interface
* **purpose**: Model-family-specific tokenization. Provider-owned — two providers for the same capability may not share a tokenizer.

=== InferenceCancellationToken
* **kind**: interface
* **purpose**: Cancellation contract for inference. Honour cancellation cooperatively at checkpoints.

=== ResponseFormat
* **kind**: sealed interface
* **purpose**: Expected response format from the provider.
* **variants**: `Text`, `Json(schema)`

=== InferenceTimeout
* **kind**: value class
* **purpose**: Records a deadline duration for timeout cancellation.

=== CancellationReason
* **kind**: sealed class
* **purpose**: Why inference was cancelled.
* **variants**: `UserRequested`, `StageTimeout`, `SessionCancelled`, `ProviderEvicted`

=== ModelLoadException
* **kind**: class
* **purpose**: Thrown when model loading fails. No event is emitted.

== event flow

*inbound:*
* `InferenceStartedEvent` → adds a new record with status `STARTED`
* `InferenceCompletedEvent` → sets status to `COMPLETED`, records tokens and latency
* `InferenceFailedEvent` → sets status to `FAILED`, records failure reason
* `InferenceTimeoutEvent` → sets status to `TIMED_OUT`, records timeout as latency

*outbound:*
* None. This module is purely a projection — events are emitted by the provider or harness.

== integration points

* `:core:events` — `StoredEvent`, inference lifecycle event types, `InferenceRequestId`, `ProviderId`, `SessionId`, `StageId`
* `:core:context` — `ContextPack` (included in `InferenceRequest`)
* `:core:artifacts` — `JsonSchema` (used in `ResponseFormat.Json`)

== invariants

* `GenerationConfig` has no implicit defaults — all relevant fields must be explicitly specified for deterministic replay
* `RoutingStrategy.select()` is a pure function with no I/O or side effects
* `DefaultInferenceRouter` never returns a provider whose health check returns `Unavailable`
* Provider IDs must be unique across the registry
* `InferenceCancellationToken` direction is one-way: a timeout can cancel the token, but cancelling the token must not retroactively signal a timeout
* `ModelLoadException` does not emit an event — model loading failure is synchronous and happens before any event would be written
* `FinishReason.Cancelled` and `InferenceStatus.TIMED_OUT` are distinct: cancellation can happen for reasons other than timeout

== PlantUML diagram

[plantuml, core-inference, "png"]
----
include::../../diagrams/core-inference.puml[]
----




== known issues

None.

== open questions

None.
