kami
9734eec63c
Generated by per-group subagents covering all 52 modules across: core (18), infrastructure (10), apps (5), interfaces (3), plugins (7), testing (9) Documentation format: - AsciiDoc (.adoc) files in docs/modules/<group>/ - PlantUML (.puml) files in docs/diagrams/ - .adoc files reference diagrams via include:: directives Each doc covers: purpose, responsibilities, non-responsibilities, key types, event flow, integration points, invariants, PlantUML diagram, known issues, and open questions.
98 lines
4.8 KiB
Plaintext
98 lines
4.8 KiB
Plaintext
= infra-inference-llama-cpp
|
|
|
|
== purpose
|
|
|
|
Implements the inference provider contracts for llama.cpp. Manages the lifecycle of a llama.cpp server subprocess, communicates with it via HTTP (Ktor client), and provides tokenization, grammar-constrained generation, and model swapping. Converts `JsonSchema` definitions to GBNF grammar for structured output.
|
|
|
|
== responsibilities
|
|
|
|
* Start, health-check, and stop a `llama-server` subprocess per model
|
|
* Expose an `InferenceProvider` that sends chat completion requests via the OpenAI-compatible HTTP API
|
|
* Provide tokenization by delegating to the llama.cpp `/tokenize` endpoint
|
|
* Convert JSON Schema to GBNF grammar strings for structured output constraints
|
|
* Emit `ModelLoadedEvent` / `ModelUnloadedEvent` on lifecycle transitions
|
|
* Enforce single-model-at-a-time via mutex
|
|
|
|
== non-responsibilities
|
|
|
|
* Does not define inference contracts — that belongs to `core:inference` and `infra:inference:commons`
|
|
* Does not manage GPU residency or scheduling
|
|
* Does not implement provider routing or fallback
|
|
* Does not cache inference results
|
|
|
|
== key types
|
|
|
|
=== LlamaCppInferenceProvider
|
|
* **kind**: class
|
|
* **purpose**: HTTP client-based `InferenceProvider` that sends chat completion requests to a llama.cpp server. Builds messages from `ContextPack`, applies GBNF grammar for JSON responses, and returns `InferenceResponse`.
|
|
* **fields**: `descriptor` (model config), `baseUrl` (default `http://127.0.0.1:10000`), `httpClient` (Ktor CIO)
|
|
|
|
=== DefaultModelManager
|
|
* **kind**: class
|
|
* **purpose**: `ModelManager` implementation that spawns and manages a `llama-server` subprocess. Thread-safe via `Mutex`. Emits lifecycle events to the event store. Supports health-check polling and process restart for model swaps.
|
|
* **fields**: `llamaServerBin` (default `"llama-server"`), `host`, `port`, `healthTimeoutMs` (default 30000), `eventStore`, `httpClient`, `eventDispatcher`
|
|
|
|
=== DefaultManagedInferenceProvider
|
|
* **kind**: class
|
|
* **purpose**: Wraps a `LlamaCppInferenceProvider` and delegates lifecycle calls to a `ModelManager`. Implements `ManagedInferenceProvider` by delegation.
|
|
|
|
=== LlamaProcess
|
|
* **kind**: class
|
|
* **purpose**: Manages the OS process lifecycle for the `llama-server` binary. Supports `start()` and `stop()` with I/O redirection to a log file.
|
|
* **fields**: `command` (process arguments), `logFile` (stdout/stderr destination)
|
|
|
|
=== LlamaCppTokenizer
|
|
* **kind**: class
|
|
* **purpose**: `Tokenizer` implementation that calls the llama.cpp `/tokenize` HTTP endpoint.
|
|
|
|
=== GbnfGrammarConverter
|
|
* **kind**: internal object
|
|
* **purpose**: Converts a `JsonSchema` (object type) to a GBNF grammar string used by llama.cpp for constrained generation. Supports required and optional keys.
|
|
|
|
=== LlamaCppApiModels
|
|
* **kind**: file-level serializable classes
|
|
* **purpose**: DTOs for the OpenAI-compatible chat completion API: `ChatCompletionRequest`, `ChatMessage`, `ChatCompletionResponse`, `Choice`, `Usage`, `TokenizeRequest`, `TokenizeResponse`.
|
|
|
|
== event flow
|
|
|
|
*Inbound:* None. Inference requests come through `InferenceProvider.infer(request)` which is called by the core inference layer.
|
|
|
|
*Outbound:*
|
|
* `ModelLoadedEvent` — emitted when a model is successfully loaded and health-checked
|
|
* `ModelUnloadedEvent` — emitted when a model is unloaded
|
|
|
|
== integration points
|
|
|
|
* `core:inference` — `InferenceProvider`, `InferenceRequest`, `InferenceResponse`, `ProviderHealth`, `Tokenizer`, `CapabilityScore`, `ResponseFormat`, `FinishReason`, `ToolCallRequest`, `ToolDefinition`
|
|
* `core:events` — `ModelLoadedEvent`, `ModelUnloadedEvent`, `ProviderId`, `SessionId`
|
|
* `core:tools` — `ToolDefinition`
|
|
* `infra:inference:commons` — `ModelManager`, `ManagedInferenceProvider`, `ModelDescriptor`, `ModelLoadException`, `ResidencyMode`
|
|
|
|
== invariants
|
|
|
|
* `DefaultModelManager` ensures single-model-at-a-time — load replaces the current model by killing and restarting the subprocess
|
|
* Health check must pass before a new model is considered loaded
|
|
* Model ID mismatch on unload throws `ModelLoadException`
|
|
* `GbnfGrammarConverter` only supports `type: "object"` schemas — other types will throw
|
|
|
|
== PlantUML diagram
|
|
|
|
[plantuml, infra-inference-llama-cpp, "png"]
|
|
----
|
|
include::../../diagrams/infra-inference-llama-cpp.puml[]
|
|
----
|
|
|
|
|
|
|
|
|
|
== known issues
|
|
|
|
* `GbnfGrammarConverter` only supports object-type schemas with string/number properties
|
|
* `DefaultManagedInferenceProvider.load()` uses `as? ManagedInferenceProvider` safe cast — returns null then throws explicit `ModelLoadException`, not `ClassCastException`
|
|
* `DefaultModelManager` has an unused `eventStore` constructor parameter (used only for `LivenessScanner` in CAS, not relevant here)
|
|
|
|
== open questions
|
|
|
|
* Should the `llama-server` binary path be configurable per-environment?
|
|
* Should process stdout/stderr be exposed for debugging beyond file logging?
|