234 lines
6.0 KiB
Markdown
234 lines
6.0 KiB
Markdown
---
|
||
name: "Core Inference Submodule Spec"
|
||
description: "Specification for :core:inference – model provider abstraction"
|
||
depth: 2
|
||
links: ["../index.md", "./core-module-spec.md", "./core-context-submodule-spec.md"]
|
||
---
|
||
|
||
# :core:inference module specification
|
||
|
||
version: 0.1-draft
|
||
status: foundational specification
|
||
|
||
---
|
||
|
||
# 1. purpose
|
||
|
||
`:core:inference` abstracts model execution behind a stable contract. It owns:
|
||
|
||
* inference provider interfaces
|
||
* inference request/response model
|
||
* model capability model
|
||
* model lifecycle management contracts
|
||
* GPU residency hints
|
||
* inference isolation
|
||
* retry/detokenization handling
|
||
|
||
Models are treated as stateless compute engines; this module ensures they are interchangeable, locally or remotely, without affecting orchestration.
|
||
|
||
---
|
||
|
||
# 2. responsibilities
|
||
|
||
`:core:inference` owns:
|
||
|
||
* `InferenceProvider` contract
|
||
* model capability definitions
|
||
* inference request schema (context pack + generation config)
|
||
* inference response schema (raw text + finish reason)
|
||
* provider registry interface
|
||
* model scheduling (which model for which stage)
|
||
* inference event definitions
|
||
* token accounting (prompt/completion tokens)
|
||
* timeout and cancellation contracts
|
||
|
||
---
|
||
|
||
# 3. non-responsibilities
|
||
|
||
`:core:inference` MUST NOT own:
|
||
|
||
* actual provider implementations (llama.cpp, ollama, etc.)
|
||
* context pack construction
|
||
* artifact parsing (that's orchestration/validation)
|
||
* model weight management (infrastructure handles storage)
|
||
* UI rendering of inference progress
|
||
|
||
---
|
||
|
||
# 4. architectural role
|
||
|
||
`:core:inference` acts as:
|
||
|
||
* the only point of contact for models
|
||
* the provider abstraction layer
|
||
* the enforcer of concurrency limits and GPU residency policies
|
||
|
||
---
|
||
|
||
# 5. design principles
|
||
|
||
## 5.1 models are stateless
|
||
|
||
Every inference call receives a full context pack. No state is retained on the model side between calls.
|
||
|
||
## 5.2 capability‑based routing
|
||
|
||
Stages request capabilities (e.g., coding, tool_calling). The inference module selects the best available model from the registry, considering local GPU budget and fallback providers.
|
||
|
||
## 5.3 deterministic‑enough configuration
|
||
|
||
Inference parameters (temperature, top_p, etc.) are part of the stage config and must be reproduced identically for replay.
|
||
|
||
## 5.4 isolation
|
||
|
||
Inference runs in a separate coroutine with strict time limits. Models may be forcibly unloaded after a swap timeout.
|
||
|
||
---
|
||
|
||
# 6. model provider contract
|
||
|
||
```kotlin
|
||
interface InferenceProvider {
|
||
val name: String
|
||
suspend fun infer(request: InferenceRequest): InferenceResponse
|
||
suspend fun healthCheck(): ProviderHealth
|
||
fun capabilities(): Set<ModelCapability>
|
||
fun tokenize(text: String): List<Token>
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
# 7. inference request/response
|
||
|
||
```kotlin
|
||
data class InferenceRequest(
|
||
val contextPack: ContextPack,
|
||
val generationConfig: GenerationConfig,
|
||
val stageId: StageId,
|
||
val sessionId: SessionId
|
||
)
|
||
|
||
data class InferenceResponse(
|
||
val text: String,
|
||
val finishReason: FinishReason,
|
||
val tokensUsed: TokenUsage,
|
||
val latencyMs: Long
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
# 8. capability negotiation
|
||
|
||
Models declare capabilities with optional scores. The inference module uses a configurable strategy (e.g., highest score first, then fallback to remote if local below threshold) to select the model for a stage.
|
||
|
||
---
|
||
|
||
# 9. model lifecycle management
|
||
|
||
Inference module defines contracts for:
|
||
|
||
* loading a model (provider specific)
|
||
* unloading
|
||
* health checks
|
||
* swap scheduling (ephemeral, dynamic, persistent residency modes)
|
||
|
||
It does not implement these; it exposes interfaces implemented by infrastructure providers.
|
||
|
||
---
|
||
|
||
# 10. inference events
|
||
|
||
`:core:inference` emits:
|
||
|
||
* `InferenceStarted`
|
||
* `InferenceCompleted`
|
||
* `InferenceFailed`
|
||
* `InferenceTimeout`
|
||
* `ModelLoaded` / `ModelUnloaded`
|
||
|
||
All carry causation back to the stage scheduling event.
|
||
|
||
---
|
||
|
||
# 11. consumed events
|
||
|
||
Consumes:
|
||
|
||
* `StageScheduled` (to know which capability to invoke)
|
||
* `ContextPackBuilt` (to receive the actual pack)
|
||
* system resource events (GPU availability)
|
||
|
||
---
|
||
|
||
# 12. invariants
|
||
|
||
* Every inference call is idempotent from the harness perspective (same context → same response… eventually, but replay uses recorded artifacts).
|
||
* Token usage always tracked.
|
||
* No cross‑session model state leakage.
|
||
|
||
---
|
||
|
||
# 13. replay semantics
|
||
|
||
During replay, inference can be skipped. Recorded `InferenceCompleted` events containing the artifact text are used instead. If inference is re‑played (e.g., for evaluation), the same config and context pack must be used.
|
||
|
||
---
|
||
|
||
# 14. threading/concurrency
|
||
|
||
Inference calls are suspending and may run on a dedicated dispatcher. Concurrency is limited by model-level `max_parallel_sessions`; the inference module enforces this limit using semaphores.
|
||
|
||
---
|
||
|
||
# 15. failure semantics
|
||
|
||
Inference failures (timeout, model crash) emit `InferenceFailed` with a typed reason. The transition engine may then retry with a different provider or fail the stage.
|
||
|
||
---
|
||
|
||
# 16. observable requirements
|
||
|
||
* inference latency
|
||
* token consumption
|
||
* model residency time
|
||
* fallback event count
|
||
* GPU utilization (via provider)
|
||
|
||
---
|
||
|
||
# 17. security boundaries
|
||
|
||
Models are untrusted. Inference runs sandboxed: no filesystem access, no network access unless explicitly configured. Artifacts are treated as untrusted text.
|
||
|
||
---
|
||
|
||
# 18. extension model
|
||
|
||
New providers are added by implementing `InferenceProvider` and registering with the provider registry. Plugins cannot alter the core inference contract.
|
||
|
||
---
|
||
|
||
# 19. forbidden patterns
|
||
|
||
* direct model state reuse between calls
|
||
* capability bypass (using a model directly without capability check)
|
||
* hidden fallback that circumvents audit
|
||
* inference without context pack
|
||
|
||
---
|
||
|
||
# 20. testing requirements
|
||
|
||
* mock provider for deterministic tests
|
||
* capability routing logic
|
||
* cancel/timeout behavior
|
||
* replay with recorded artifacts
|
||
|
||
---
|
||
|
||
# 21. philosophy
|
||
|
||
Inference is a utility, not a partner. This module treats language models like an accelerator card: it submits a well‑formed job, waits for a result, and never assumes the result is anything but a candidate. |