kami
9734eec63c
Generated by per-group subagents covering all 52 modules across: core (18), infrastructure (10), apps (5), interfaces (3), plugins (7), testing (9) Documentation format: - AsciiDoc (.adoc) files in docs/modules/<group>/ - PlantUML (.puml) files in docs/diagrams/ - .adoc files reference diagrams via include:: directives Each doc covers: purpose, responsibilities, non-responsibilities, key types, event flow, integration points, invariants, PlantUML diagram, known issues, and open questions.
186 lines
7.3 KiB
Plaintext
186 lines
7.3 KiB
Plaintext
= core-inference
|
||
|
||
== purpose
|
||
|
||
Abstracts LLM inference behind a provider interface, routes requests to healthy providers based on required capabilities, tracks inference lifecycle (started → completed/failed/timed-out) as event-sourced state, and defines the data types used to communicate with inference providers (requests, responses, tool definitions, generation config).
|
||
|
||
== responsibilities
|
||
|
||
* Defining the `InferenceProvider` contract for LLM inference calls
|
||
* Routing inference requests to the most suitable healthy provider via `InferenceRouter`
|
||
* Caching provider health checks with configurable TTL
|
||
* Tracking inference records and their status transitions
|
||
* Defining shared inference types (`InferenceRequest`, `InferenceResponse`, `GenerationConfig`, `ToolDefinition`, etc.)
|
||
* Providing tokenizer abstraction for model-family-specific tokenization
|
||
* Defining model capabilities (`Coding`, `ToolCalling`, `Reasoning`, etc.) for provider selection
|
||
|
||
== non-responsibilities
|
||
|
||
* Does not implement concrete providers — those live in `:infrastructure:*`
|
||
* Does not manage context construction — uses `ContextPack` from `:core:context`
|
||
* Does not handle tool execution — only models tool call declarations for the LLM
|
||
* Does not emit events for model loading failures — those are synchronous exceptions
|
||
|
||
== key types
|
||
|
||
=== InferenceProvider
|
||
* **kind**: interface
|
||
* **purpose**: Contract for a single LLM provider. Can infer, health-check, and report capabilities.
|
||
|
||
=== ProviderRegistry
|
||
* **kind**: interface
|
||
* **purpose**: Manages the set of registered `InferenceProvider`s. Supports capability-based resolution.
|
||
|
||
=== InferenceRouter
|
||
* **kind**: interface
|
||
* **purpose**: Resolves a provider for a given stage and set of required capabilities. Performs live health checks before selection.
|
||
|
||
=== DefaultInferenceRouter
|
||
* **kind**: class
|
||
* **purpose**: Uses a `RoutingStrategy` to select from healthy candidates. Caches health results with configurable TTL. Re-checks health post-selection to close TOCTOU window.
|
||
|
||
=== RoutingStrategy
|
||
* **kind**: fun interface
|
||
* **purpose**: Pure selection function — chooses a provider from candidates given required capabilities. Throws `NoEligibleProviderException` if no candidate qualifies.
|
||
|
||
=== InferenceRequest
|
||
* **kind**: data class
|
||
* **purpose**: Complete input for an LLM inference call. Includes context pack, generation config, tool definitions, and response format.
|
||
* **fields**: `requestId`, `sessionId`, `stageId`, `contextPack`, `generationConfig`, `timeout`, `responseFormat`, `tools`
|
||
|
||
=== InferenceResponse
|
||
* **kind**: data class
|
||
* **purpose**: Output of an LLM inference call.
|
||
* **fields**: `requestId`, `text`, `finishReason`, `tokensUsed`, `latencyMs`, `toolCalls`
|
||
|
||
=== GenerationConfig
|
||
* **kind**: data class
|
||
* **purpose**: Parameters for LLM generation — all fields required for deterministic replay.
|
||
* **fields**: `temperature`, `topP`, `maxTokens`, `stopSequences`, `seed`
|
||
|
||
=== FinishReason
|
||
* **kind**: sealed class
|
||
* **purpose**: Why inference finished.
|
||
* **variants**: `Stop`, `Length`, `ToolCall`, `Timeout`, `Cancelled`, `Error(message)`
|
||
|
||
=== ToolCallRequest
|
||
* **kind**: data class
|
||
* **purpose**: A tool call proposed by the LLM.
|
||
* **fields**: `function` (ToolCallFunction with name and arguments)
|
||
|
||
=== ToolDefinition
|
||
* **kind**: data class
|
||
* **purpose**: A tool declaration sent to the LLM as part of an inference request.
|
||
* **fields**: `function` (ToolFunction with name, description, parameters)
|
||
|
||
=== ModelCapability
|
||
* **kind**: sealed class
|
||
* **purpose**: Broad capability category for provider selection.
|
||
* **variants**: `Coding`, `ToolCalling`, `Reasoning`, `Summarization`, `General`
|
||
|
||
=== CapabilityScore
|
||
* **kind**: data class
|
||
* **purpose**: Scores a provider on a single capability (0.0–1.0).
|
||
|
||
=== ProviderHealth
|
||
* **kind**: sealed class
|
||
* **purpose**: Health status of a provider.
|
||
* **variants**: `Healthy`, `Degraded(reason)`, `Unavailable(reason)`
|
||
|
||
=== InferenceState
|
||
* **kind**: data class
|
||
* **purpose**: Event-sourced projection of all inference records for a session.
|
||
* **fields**: `records` (list of `InferenceRecord`)
|
||
|
||
=== InferenceRecord
|
||
* **kind**: data class
|
||
* **purpose**: Immutable record of a single inference call.
|
||
* **fields**: `requestId`, `sessionId`, `stageId`, `providerId`, `status`, `tokensUsed`, `latencyMs`, `failureReason`
|
||
|
||
=== InferenceStatus
|
||
* **kind**: enum
|
||
* **variants**: `STARTED`, `COMPLETED`, `FAILED`, `TIMED_OUT`
|
||
|
||
=== InferenceReducer / DefaultInferenceReducer
|
||
* **kind**: interface / class
|
||
* **purpose**: Transforms `InferenceState` given stored events. Handles `InferenceStartedEvent`, `InferenceCompletedEvent`, `InferenceFailedEvent`, `InferenceTimeoutEvent`.
|
||
|
||
=== InferenceProjector
|
||
* **kind**: class
|
||
* **purpose**: Wraps `InferenceReducer` as a `Projection<InferenceState>`.
|
||
|
||
=== InferenceRepository
|
||
* **kind**: class
|
||
* **purpose**: Rebuilds `InferenceState` for a session via `EventReplayer`.
|
||
|
||
=== Tokenizer
|
||
* **kind**: interface
|
||
* **purpose**: Model-family-specific tokenization. Provider-owned — two providers for the same capability may not share a tokenizer.
|
||
|
||
=== InferenceCancellationToken
|
||
* **kind**: interface
|
||
* **purpose**: Cancellation contract for inference. Honour cancellation cooperatively at checkpoints.
|
||
|
||
=== ResponseFormat
|
||
* **kind**: sealed interface
|
||
* **purpose**: Expected response format from the provider.
|
||
* **variants**: `Text`, `Json(schema)`
|
||
|
||
=== InferenceTimeout
|
||
* **kind**: value class
|
||
* **purpose**: Records a deadline duration for timeout cancellation.
|
||
|
||
=== CancellationReason
|
||
* **kind**: sealed class
|
||
* **purpose**: Why inference was cancelled.
|
||
* **variants**: `UserRequested`, `StageTimeout`, `SessionCancelled`, `ProviderEvicted`
|
||
|
||
=== ModelLoadException
|
||
* **kind**: class
|
||
* **purpose**: Thrown when model loading fails. No event is emitted.
|
||
|
||
== event flow
|
||
|
||
*inbound:*
|
||
* `InferenceStartedEvent` → adds a new record with status `STARTED`
|
||
* `InferenceCompletedEvent` → sets status to `COMPLETED`, records tokens and latency
|
||
* `InferenceFailedEvent` → sets status to `FAILED`, records failure reason
|
||
* `InferenceTimeoutEvent` → sets status to `TIMED_OUT`, records timeout as latency
|
||
|
||
*outbound:*
|
||
* None. This module is purely a projection — events are emitted by the provider or harness.
|
||
|
||
== integration points
|
||
|
||
* `:core:events` — `StoredEvent`, inference lifecycle event types, `InferenceRequestId`, `ProviderId`, `SessionId`, `StageId`
|
||
* `:core:context` — `ContextPack` (included in `InferenceRequest`)
|
||
* `:core:artifacts` — `JsonSchema` (used in `ResponseFormat.Json`)
|
||
|
||
== invariants
|
||
|
||
* `GenerationConfig` has no implicit defaults — all relevant fields must be explicitly specified for deterministic replay
|
||
* `RoutingStrategy.select()` is a pure function with no I/O or side effects
|
||
* `DefaultInferenceRouter` never returns a provider whose health check returns `Unavailable`
|
||
* Provider IDs must be unique across the registry
|
||
* `InferenceCancellationToken` direction is one-way: a timeout can cancel the token, but cancelling the token must not retroactively signal a timeout
|
||
* `ModelLoadException` does not emit an event — model loading failure is synchronous and happens before any event would be written
|
||
* `FinishReason.Cancelled` and `InferenceStatus.TIMED_OUT` are distinct: cancellation can happen for reasons other than timeout
|
||
|
||
== PlantUML diagram
|
||
|
||
[plantuml, core-inference, "png"]
|
||
----
|
||
include::../../diagrams/core-inference.puml[]
|
||
----
|
||
|
||
|
||
|
||
|
||
== known issues
|
||
|
||
None.
|
||
|
||
== open questions
|
||
|
||
None.
|