= infra-inference-llama-cpp

== purpose

Implements the inference provider contracts for llama.cpp. Manages the lifecycle of a llama.cpp server subprocess, communicates with it via HTTP (Ktor client), and provides tokenization, grammar-constrained generation, and model swapping. Converts `JsonSchema` definitions to GBNF grammar for structured output.

== responsibilities

* Start, health-check, and stop a `llama-server` subprocess per model
* Expose an `InferenceProvider` that sends chat completion requests via the OpenAI-compatible HTTP API
* Provide tokenization by delegating to the llama.cpp `/tokenize` endpoint
* Convert JSON Schema to GBNF grammar strings for structured output constraints
* Emit `ModelLoadedEvent` / `ModelUnloadedEvent` on lifecycle transitions
* Enforce single-model-at-a-time via mutex

== non-responsibilities

* Does not define inference contracts — that belongs to `core:inference` and `infra:inference:commons`
* Does not manage GPU residency or scheduling
* Does not implement provider routing or fallback
* Does not cache inference results

== key types

=== LlamaCppInferenceProvider
* **kind**: class
* **purpose**: HTTP client-based `InferenceProvider` that sends chat completion requests to a llama.cpp server. Builds messages from `ContextPack`, applies GBNF grammar for JSON responses, and returns `InferenceResponse`.
* **fields**: `descriptor` (model config), `baseUrl` (default `http://127.0.0.1:10000`), `httpClient` (Ktor CIO)

=== DefaultModelManager
* **kind**: class
* **purpose**: `ModelManager` implementation that spawns and manages a `llama-server` subprocess. Thread-safe via `Mutex`. Emits lifecycle events to the event store. Supports health-check polling and process restart for model swaps.
* **fields**: `llamaServerBin` (default `"llama-server"`), `host`, `port`, `healthTimeoutMs` (default 30000), `eventStore`, `httpClient`, `eventDispatcher`

=== DefaultManagedInferenceProvider
* **kind**: class
* **purpose**: Wraps a `LlamaCppInferenceProvider` and delegates lifecycle calls to a `ModelManager`. Implements `ManagedInferenceProvider` by delegation.

=== LlamaProcess
* **kind**: class
* **purpose**: Manages the OS process lifecycle for the `llama-server` binary. Supports `start()` and `stop()` with I/O redirection to a log file.
* **fields**: `command` (process arguments), `logFile` (stdout/stderr destination)

=== LlamaCppTokenizer
* **kind**: class
* **purpose**: `Tokenizer` implementation that calls the llama.cpp `/tokenize` HTTP endpoint.

=== GbnfGrammarConverter
* **kind**: internal object
* **purpose**: Converts a `JsonSchema` (object type) to a GBNF grammar string used by llama.cpp for constrained generation. Supports required and optional keys.

=== LlamaCppApiModels
* **kind**: file-level serializable classes
* **purpose**: DTOs for the OpenAI-compatible chat completion API: `ChatCompletionRequest`, `ChatMessage`, `ChatCompletionResponse`, `Choice`, `Usage`, `TokenizeRequest`, `TokenizeResponse`.

== event flow

*Inbound:* None. Inference requests come through `InferenceProvider.infer(request)` which is called by the core inference layer.

*Outbound:*
* `ModelLoadedEvent` — emitted when a model is successfully loaded and health-checked
* `ModelUnloadedEvent` — emitted when a model is unloaded

== integration points

* `core:inference` — `InferenceProvider`, `InferenceRequest`, `InferenceResponse`, `ProviderHealth`, `Tokenizer`, `CapabilityScore`, `ResponseFormat`, `FinishReason`, `ToolCallRequest`, `ToolDefinition`
* `core:events` — `ModelLoadedEvent`, `ModelUnloadedEvent`, `ProviderId`, `SessionId`
* `core:tools` — `ToolDefinition`
* `infra:inference:commons` — `ModelManager`, `ManagedInferenceProvider`, `ModelDescriptor`, `ModelLoadException`, `ResidencyMode`

== invariants

* `DefaultModelManager` ensures single-model-at-a-time — load replaces the current model by killing and restarting the subprocess
* Health check must pass before a new model is considered loaded
* Model ID mismatch on unload throws `ModelLoadException`
* `GbnfGrammarConverter` only supports `type: "object"` schemas — other types will throw

== PlantUML diagram

[plantuml, infra-inference-llama-cpp, "png"]
----
include::../../diagrams/infra-inference-llama-cpp.puml[]
----




== known issues

* `GbnfGrammarConverter` only supports object-type schemas with string/number properties
* `DefaultManagedInferenceProvider.load()` uses `as? ManagedInferenceProvider` safe cast — returns null then throws explicit `ModelLoadException`, not `ClassCastException`
* `DefaultModelManager` has an unused `eventStore` constructor parameter (used only for `LivenessScanner` in CAS, not relevant here)

== open questions

* Should the `llama-server` binary path be configurable per-environment?
* Should process stdout/stderr be exposed for debugging beyond file logging?