correx/docs/modules/core-inference-submodule-spec.md

name, description, depth, links

name	description	depth	links
Core Inference Submodule Spec	Specification for :core:inference – model provider abstraction	2	../index.md ./core-module-spec.md ./core-context-submodule-spec.md

:core:inference module specification

version: 0.1-draft status: foundational specification

---

1. purpose

:core:inference abstracts model execution behind a stable contract. It owns:

• inference provider interfaces
• inference request/response model
• model capability model
• model lifecycle management contracts
• GPU residency hints
• inference isolation
• retry/detokenization handling

Models are treated as stateless compute engines; this module ensures they are interchangeable, locally or remotely, without affecting orchestration.

---

2. responsibilities

:core:inference owns:

• InferenceProvider contract
• model capability definitions
• inference request schema (context pack + generation config)
• inference response schema (raw text + finish reason)
• provider registry interface
• model scheduling (which model for which stage)
• inference event definitions
• token accounting (prompt/completion tokens)
• timeout and cancellation contracts

---

3. non-responsibilities

:core:inference MUST NOT own:

• actual provider implementations (llama.cpp, ollama, etc.)
• context pack construction
• artifact parsing (that's orchestration/validation)
• model weight management (infrastructure handles storage)
• UI rendering of inference progress

---

4. architectural role

:core:inference acts as:

• the only point of contact for models
• the provider abstraction layer
• the enforcer of concurrency limits and GPU residency policies

---

5. design principles

5.1 models are stateless

Every inference call receives a full context pack. No state is retained on the model side between calls.

5.2 capability‑based routing

Stages request capabilities (e.g., coding, tool_calling). The inference module selects the best available model from the registry, considering local GPU budget and fallback providers.

5.3 deterministic‑enough configuration

Inference parameters (temperature, top_p, etc.) are part of the stage config and must be reproduced identically for replay.

5.4 isolation

Inference runs in a separate coroutine with strict time limits. Models may be forcibly unloaded after a swap timeout.

---

6. model provider contract

interface InferenceProvider {
    val name: String
    suspend fun infer(request: InferenceRequest): InferenceResponse
    suspend fun healthCheck(): ProviderHealth
    fun capabilities(): Set<ModelCapability>
    fun tokenize(text: String): List<Token>
}

---

7. inference request/response

data class InferenceRequest(
    val contextPack: ContextPack,
    val generationConfig: GenerationConfig,
    val stageId: StageId,
    val sessionId: SessionId
)

data class InferenceResponse(
    val text: String,
    val finishReason: FinishReason,
    val tokensUsed: TokenUsage,
    val latencyMs: Long
)

---

8. capability negotiation

Models declare capabilities with optional scores. The inference module uses a configurable strategy (e.g., highest score first, then fallback to remote if local below threshold) to select the model for a stage.

---

9. model lifecycle management

Inference module defines contracts for:

• loading a model (provider specific)
• unloading
• health checks
• swap scheduling (ephemeral, dynamic, persistent residency modes)

It does not implement these; it exposes interfaces implemented by infrastructure providers.

---

10. inference events

:core:inference emits:

• InferenceStarted
• InferenceCompleted
• InferenceFailed
• InferenceTimeout
• ModelLoaded / ModelUnloaded

All carry causation back to the stage scheduling event.

---

11. consumed events

Consumes:

• StageScheduled (to know which capability to invoke)
• ContextPackBuilt (to receive the actual pack)
• system resource events (GPU availability)

---

12. invariants

• Every inference call is idempotent from the harness perspective (same context → same response… eventually, but replay uses recorded artifacts).
• Token usage always tracked.
• No cross‑session model state leakage.

---

13. replay semantics

During replay, inference can be skipped. Recorded InferenceCompleted events containing the artifact text are used instead. If inference is re‑played (e.g., for evaluation), the same config and context pack must be used.

---

14. threading/concurrency

Inference calls are suspending and may run on a dedicated dispatcher. Concurrency is limited by model-level max_parallel_sessions; the inference module enforces this limit using semaphores.

---

15. failure semantics

Inference failures (timeout, model crash) emit InferenceFailed with a typed reason. The transition engine may then retry with a different provider or fail the stage.

---

16. observable requirements

• inference latency
• token consumption
• model residency time
• fallback event count
• GPU utilization (via provider)

---

17. security boundaries

Models are untrusted. Inference runs sandboxed: no filesystem access, no network access unless explicitly configured. Artifacts are treated as untrusted text.

---

18. extension model

New providers are added by implementing InferenceProvider and registering with the provider registry. Plugins cannot alter the core inference contract.

---

19. forbidden patterns

• direct model state reuse between calls
• capability bypass (using a model directly without capability check)
• hidden fallback that circumvents audit
• inference without context pack

---

20. testing requirements

• mock provider for deterministic tests
• capability routing logic
• cancel/timeout behavior
• replay with recorded artifacts

---

21. philosophy

Inference is a utility, not a partner. This module treats language models like an accelerator card: it submits a well‑formed job, waits for a result, and never assumes the result is anything but a candidate.