---
name: "Core Inference Submodule Spec"
description: "Specification for :core:inference – model provider abstraction"
depth: 2
links: ["../index.md", "./core-module-spec.md", "./core-context-submodule-spec.md"]
---

# :core:inference module specification

version: 0.1-draft
status: foundational specification

---

# 1. purpose

`:core:inference` abstracts model execution behind a stable contract. It owns:

* inference provider interfaces
* inference request/response model
* model capability model
* model lifecycle management contracts
* GPU residency hints
* inference isolation
* retry/detokenization handling

Models are treated as stateless compute engines; this module ensures they are interchangeable, locally or remotely, without affecting orchestration.

---

# 2. responsibilities

`:core:inference` owns:

* `InferenceProvider` contract
* model capability definitions
* inference request schema (context pack + generation config)
* inference response schema (raw text + finish reason)
* provider registry interface
* model scheduling (which model for which stage)
* inference event definitions
* token accounting (prompt/completion tokens)
* timeout and cancellation contracts

---

# 3. non-responsibilities

`:core:inference` MUST NOT own:

* actual provider implementations (llama.cpp, ollama, etc.)
* context pack construction
* artifact parsing (that's orchestration/validation)
* model weight management (infrastructure handles storage)
* UI rendering of inference progress

---

# 4. architectural role

`:core:inference` acts as:

* the only point of contact for models
* the provider abstraction layer
* the enforcer of concurrency limits and GPU residency policies

---

# 5. design principles

## 5.1 models are stateless

Every inference call receives a full context pack. No state is retained on the model side between calls.

## 5.2 capability‑based routing

Stages request capabilities (e.g., coding, tool_calling). The inference module selects the best available model from the registry, considering local GPU budget and fallback providers.

## 5.3 deterministic‑enough configuration

Inference parameters (temperature, top_p, etc.) are part of the stage config and must be reproduced identically for replay.

## 5.4 isolation

Inference runs in a separate coroutine with strict time limits. Models may be forcibly unloaded after a swap timeout.

---

# 6. model provider contract

```kotlin
interface InferenceProvider {
    val name: String
    suspend fun infer(request: InferenceRequest): InferenceResponse
    suspend fun healthCheck(): ProviderHealth
    fun capabilities(): Set<ModelCapability>
    fun tokenize(text: String): List<Token>
}
```

---

# 7. inference request/response

```kotlin
data class InferenceRequest(
    val contextPack: ContextPack,
    val generationConfig: GenerationConfig,
    val stageId: StageId,
    val sessionId: SessionId
)

data class InferenceResponse(
    val text: String,
    val finishReason: FinishReason,
    val tokensUsed: TokenUsage,
    val latencyMs: Long
)
```

---

# 8. capability negotiation

Models declare capabilities with optional scores. The inference module uses a configurable strategy (e.g., highest score first, then fallback to remote if local below threshold) to select the model for a stage.

---

# 9. model lifecycle management

Inference module defines contracts for:

* loading a model (provider specific)
* unloading
* health checks
* swap scheduling (ephemeral, dynamic, persistent residency modes)

It does not implement these; it exposes interfaces implemented by infrastructure providers.

---

# 10. inference events

`:core:inference` emits:

* `InferenceStarted`
* `InferenceCompleted`
* `InferenceFailed`
* `InferenceTimeout`
* `ModelLoaded` / `ModelUnloaded`

All carry causation back to the stage scheduling event.

---

# 11. consumed events

Consumes:

* `StageScheduled` (to know which capability to invoke)
* `ContextPackBuilt` (to receive the actual pack)
* system resource events (GPU availability)

---

# 12. invariants

* Every inference call is idempotent from the harness perspective (same context → same response… eventually, but replay uses recorded artifacts).
* Token usage always tracked.
* No cross‑session model state leakage.

---

# 13. replay semantics

During replay, inference can be skipped. Recorded `InferenceCompleted` events containing the artifact text are used instead. If inference is re‑played (e.g., for evaluation), the same config and context pack must be used.

---

# 14. threading/concurrency

Inference calls are suspending and may run on a dedicated dispatcher. Concurrency is limited by model-level `max_parallel_sessions`; the inference module enforces this limit using semaphores.

---

# 15. failure semantics

Inference failures (timeout, model crash) emit `InferenceFailed` with a typed reason. The transition engine may then retry with a different provider or fail the stage.

---

# 16. observable requirements

* inference latency
* token consumption
* model residency time
* fallback event count
* GPU utilization (via provider)

---

# 17. security boundaries

Models are untrusted. Inference runs sandboxed: no filesystem access, no network access unless explicitly configured. Artifacts are treated as untrusted text.

---

# 18. extension model

New providers are added by implementing `InferenceProvider` and registering with the provider registry. Plugins cannot alter the core inference contract.

---

# 19. forbidden patterns

* direct model state reuse between calls
* capability bypass (using a model directly without capability check)
* hidden fallback that circumvents audit
* inference without context pack

---

# 20. testing requirements

* mock provider for deterministic tests
* capability routing logic
* cancel/timeout behavior
* replay with recorded artifacts

---

# 21. philosophy

Inference is a utility, not a partner. This module treats language models like an accelerator card: it submits a well‑formed job, waits for a result, and never assumes the result is anything but a candidate.