kami
9734eec63c
Generated by per-group subagents covering all 52 modules across: core (18), infrastructure (10), apps (5), interfaces (3), plugins (7), testing (9) Documentation format: - AsciiDoc (.adoc) files in docs/modules/<group>/ - PlantUML (.puml) files in docs/diagrams/ - .adoc files reference diagrams via include:: directives Each doc covers: purpose, responsibilities, non-responsibilities, key types, event flow, integration points, invariants, PlantUML diagram, known issues, and open questions.
79 lines
3.1 KiB
Plaintext
79 lines
3.1 KiB
Plaintext
= infra-inference-commons
|
|
|
|
== purpose
|
|
|
|
Defines shared abstractions and types for inference provider lifecycle management. Provides the `ModelManager` interface for loading/unloading models, the `ManagedInferenceProvider` wrapper that ties an `InferenceProvider` to a model lifecycle, and supporting types (`ModelDescriptor`, `ResidencyMode`, `ModelLoadException`).
|
|
|
|
== responsibilities
|
|
|
|
* Define the contract for model lifecycle management (load, unload, health check)
|
|
* Provide a managed wrapper around `InferenceProvider` that supports model replacement
|
|
* Define residency modes that govern when models are unloaded (persistent, dynamic, ephemeral)
|
|
* Describe model configuration via `ModelDescriptor`
|
|
|
|
== non-responsibilities
|
|
|
|
* Does not implement any inference provider — that belongs to `infra:inference:llama-cpp` and similar
|
|
* Does not handle provider routing or selection
|
|
* Does not define the core inference contract — that belongs to `core:inference`
|
|
|
|
== key types
|
|
|
|
=== ModelManager
|
|
* **kind**: interface
|
|
* **purpose**: Contract for loading, unloading, and health-checking inference models. Enforces single-model-at-a-time.
|
|
* **methods**: `load(descriptor)`, `unload(modelId)`, `currentModel()`, `healthCheck()`
|
|
|
|
=== ManagedInferenceProvider
|
|
* **kind**: interface
|
|
* **purpose**: Extends `InferenceProvider` with lifecycle management (unload, isLoaded, load a different model).
|
|
* **methods**: `unload()`, `isLoaded()`, `load(newDescriptor)`
|
|
|
|
=== ModelDescriptor
|
|
* **kind**: data class
|
|
* **purpose**: Describes a model instance with its path, residency mode, context size, and capabilities.
|
|
* **fields**: `modelId`, `modelPath`, `residencyMode`, `idleTimeoutMs` (default 60000), `contextSize` (default 8192), `capabilities`
|
|
|
|
=== ResidencyMode
|
|
* **kind**: enum class
|
|
* **purpose**: Governs model unloading behavior.
|
|
* **variants**: `PERSISTENT` (never unload), `DYNAMIC` (unload after idle timeout), `EPHEMERAL` (unload immediately after inference)
|
|
|
|
=== ModelLoadException
|
|
* **kind**: class
|
|
* **purpose**: Exception thrown when model loading or unloading fails. Carries `modelId` and optional cause.
|
|
* **fields**: `message`, `modelId`, `cause`
|
|
|
|
== event flow
|
|
|
|
*None.* The commons layer defines abstractions only; events are emitted by implementations.
|
|
|
|
== integration points
|
|
|
|
* `core:inference` — `InferenceProvider`, `ProviderHealth`, `CapabilityScore`
|
|
|
|
== invariants
|
|
|
|
* `ModelManager` enforces single-model-at-a-time via mutex in implementations
|
|
* Model loading must succeed through health checks before the provider is returned
|
|
* `ManagedInferenceProvider.load()` replaces the current model entirely
|
|
|
|
== PlantUML diagram
|
|
|
|
[plantuml, infra-inference-commons, "png"]
|
|
----
|
|
include::../../diagrams/infra-inference-commons.puml[]
|
|
----
|
|
|
|
|
|
|
|
|
|
== known issues
|
|
|
|
* `ManagedInferenceProvider.load()` uses `as? ManagedInferenceProvider` safe cast — returns null then throws explicit `ModelLoadException`, not `ClassCastException`
|
|
|
|
== open questions
|
|
|
|
* Should `ModelManager` support multiple concurrent models for different sessions?
|
|
* Should `ResidencyMode.DYNAMIC` have a configurable timeout per-model or per-manager?
|