195 lines
7.7 KiB
Markdown
195 lines
7.7 KiB
Markdown
# Epic 9 — Inference Abstraction (:core:inference)
|
||
|
||
## completed deliverables
|
||
|
||
### 1. core value types
|
||
|
||
introduced the full set of inference-domain value types as the semantic foundation of the module.
|
||
|
||
final structures:
|
||
|
||
* `ModelCapability` — sealed class (Coding, ToolCalling, Reasoning, Summarization, General)
|
||
* `CapabilityScore` — capability + score (0.0–1.0); used for provider selection
|
||
* `FinishReason` — sealed: Stop, Length, Timeout, Cancelled, Error(message)
|
||
* `ProviderHealth` — sealed: Healthy, Degraded(reason), Unavailable(reason)
|
||
* `CancellationReason` — sealed: UserRequested, StageTimeout, SessionCancelled, ProviderEvicted
|
||
* `InferenceTimeout` — value class wrapping Duration
|
||
* `TokenUsage` — promptTokens, completionTokens, totalTokens (derived)
|
||
* `GenerationConfig` — fully serializable; temperature, topP, maxTokens, stopSequences, seed
|
||
|
||
key properties:
|
||
|
||
* `GenerationConfig` and `TokenUsage` are `@Serializable` — replay requirement
|
||
* `ProviderHealth` and `CancellationReason` are runtime-only sealed classes — not serialized directly
|
||
* events carry string labels for cancellation context, not sealed class instances
|
||
|
||
---
|
||
|
||
### 2. inference request / response model
|
||
|
||
introduced the canonical data contract for all inference calls in the system.
|
||
|
||
final structures:
|
||
|
||
* `InferenceRequestId` — opaque value class; caller-generated, provider treats as pass-through
|
||
* `InferenceRequest` — requestId + sessionId + stageId + contextPack + generationConfig + optional timeout
|
||
* `InferenceResponse` — requestId + text + finishReason + tokensUsed + latencyMs
|
||
|
||
both are `@Serializable` for replay correctness.
|
||
|
||
`requestId` is always caller-generated — consistent with identity ownership across sessions, artifacts, and approvals modules.
|
||
|
||
---
|
||
|
||
### 3. tokenizer interface
|
||
|
||
introduced a provider-owned tokenizer abstraction, separated from the provider interface itself.
|
||
|
||
final structures:
|
||
|
||
* `Token` — value class wrapping Int
|
||
* `Tokenizer` — interface: `tokenize(text): List<Token>`, `countTokens(text): Int`
|
||
|
||
tokenizer is a property of `InferenceProvider`, not a method — makes it clear that tokenization is a static capability of the model, not a per-call operation. two providers declaring the same `ModelCapability` may not share a tokenizer.
|
||
|
||
---
|
||
|
||
### 4. provider interface and registry
|
||
|
||
introduced the core provider abstraction and its lookup contract.
|
||
|
||
final structures:
|
||
|
||
* `ProviderId` — value class
|
||
* `InferenceProvider` — interface: id, name, tokenizer, infer, healthCheck, capabilities
|
||
* `ProviderRegistry` — interface: register, resolve(capability), listAll, healthCheckAll
|
||
|
||
`resolve(capability)` returns an ordered list (score descending) — routing strategy picks from this list; registry does not select.
|
||
|
||
---
|
||
|
||
### 5. routing contracts
|
||
|
||
introduced pluggable routing as a first-class contract, not hardcoded selection logic.
|
||
|
||
final structures:
|
||
|
||
* `RoutingStrategy` — interface: `select(candidates, requiredCapabilities): InferenceProvider`
|
||
* `InferenceRouter` — interface: `route(stageId, requiredCapabilities): InferenceProvider`
|
||
* `NoEligibleProviderException` — typed failure; no nullable returns
|
||
|
||
routing is pure — no I/O, operates on a snapshot of available providers. decoupling `RoutingStrategy` from `InferenceRouter` allows selection logic (best score, round-robin, fallback chain) to be swapped without touching the router contract.
|
||
|
||
---
|
||
|
||
### 6. cancellation semantics
|
||
|
||
introduced the cancellation contract and established the coroutine cooperation rules for all provider implementations.
|
||
|
||
final structure:
|
||
|
||
* `InferenceCancellationToken` — interface: isCancelled, cancel(reason)
|
||
|
||
coroutine contract (documented as KDoc on the interface):
|
||
|
||
* `infer()` must call `ensureActive()` before the socket write, after each streamed chunk, and before parsing the final response
|
||
* blocking calls must be wrapped with `withContext(Dispatchers.IO)`
|
||
* `CancellationException` must never be swallowed
|
||
|
||
contract is intentionally documentation-only at this layer — enforcement lives in implementations (epic 11).
|
||
|
||
---
|
||
|
||
### 7. inference events and serialization module
|
||
|
||
introduced the full set of inference lifecycle events and registered them in the polymorphic serialization system.
|
||
|
||
events:
|
||
|
||
* `InferenceStartedEvent` — requestId, sessionId, stageId, providerId
|
||
* `InferenceCompletedEvent` — requestId, sessionId, stageId, providerId, tokensUsed, latencyMs
|
||
* `InferenceFailedEvent` — requestId, sessionId, stageId, providerId, reason (string)
|
||
* `InferenceTimeoutEvent` — requestId, sessionId, stageId, providerId, timeoutMs
|
||
* `ModelLoadedEvent` — providerId, sessionId
|
||
* `ModelUnloadedEvent` — providerId, sessionId, cancellationReason (string, optional)
|
||
|
||
all events carry `requestId` for causation tracing back to the originating call.
|
||
|
||
`inferenceModule: SerializersModule` registers all payloads — follows the same pattern as `eventModule`, `artifactModule`, `contextModule`.
|
||
|
||
---
|
||
|
||
### 8. mock provider and contract test infrastructure
|
||
|
||
introduced a deterministic fake provider for contract validation and test isolation.
|
||
|
||
final structures:
|
||
|
||
* `MockTokenizer` — character-based approximation (1 token ≈ 4 chars); deterministic; not model-accurate
|
||
* `MockInferenceProvider` — configurable: fixed response, artificial delay, forced failure; exposes `inferCallCount` for assertion
|
||
* `InferenceProviderContractTest` — abstract contract test class; same shape as `EventStoreContractTest`
|
||
|
||
contract tests cover:
|
||
|
||
* response carries matching requestId
|
||
* text is non-blank
|
||
* token usage is tracked and positive
|
||
* latency is non-negative
|
||
* healthCheck returns non-null
|
||
* capabilities non-empty with scores in 0.0–1.0
|
||
* coroutine cancellation is respected
|
||
* tokenizer count is consistent with tokenize
|
||
|
||
---
|
||
|
||
# final architecture after Epic 9
|
||
|
||
```text
|
||
InferenceRouter (capability-driven routing)
|
||
↓
|
||
RoutingStrategy (pluggable selection logic)
|
||
↓
|
||
ProviderRegistry (capability → provider resolution)
|
||
↓
|
||
InferenceProvider (stateless compute contract)
|
||
├── tokenizer: Tokenizer
|
||
├── infer(InferenceRequest): InferenceResponse
|
||
└── capabilities(): Set<CapabilityScore>
|
||
↓
|
||
InferenceEvents → EventStore (via core:events)
|
||
```
|
||
|
||
---
|
||
|
||
# major architectural outcomes
|
||
|
||
Epic 9 established:
|
||
|
||
* stable inference provider abstraction — models are interchangeable, infrastructure-independent
|
||
* capability-based routing with pluggable selection strategy
|
||
* provider-owned tokenization — no shared tokenizer assumption across model families
|
||
* caller-generated request identity — consistent with system-wide identity ownership model
|
||
* cooperative cancellation contract — enforced at implementation layer, documented at contract layer
|
||
* full inference lifecycle event coverage with causation tracing via requestId
|
||
* replay-safe serialization for all request/response/config types
|
||
* mock provider and abstract contract tests for deterministic validation in all future epics
|
||
|
||
---
|
||
|
||
# what Epic 9 intentionally does NOT include
|
||
|
||
not implemented:
|
||
|
||
* inference state projection / reducer / repository — deferred to Epic 10 where the orchestrator provides a concrete consumer
|
||
* actual provider implementations (llama.cpp, ollama) — infrastructure concern, Epic 11
|
||
* streaming response handling — deferred; contract is text-final only for now
|
||
* GPU residency scheduling — infrastructure concern, Epic 11
|
||
* router context isolation (separate L2 memory per provider) — deferred to Epic 13
|
||
|
||
---
|
||
|
||
# final state
|
||
|
||
`:core:inference` provides:
|
||
|
||
> a fully specified, infrastructure-independent inference abstraction layer with capability-based routing, cooperative cancellation semantics, replay-safe request/response contracts, and a deterministic mock provider — ready to be composed by the orchestration kernel in Epic 10. |