feat(server,cli): event-sourced health checks — disk watermark probe (observability §4)
Health states are events. A HealthMonitor polls registered HealthProbes on an interval and appends HealthDegraded/HealthRestored only on a status transition (hysteresis), so health is observable, replayable, and renderable like every other fact in the log. The probe reads nondeterministic environment state and records it at observe-time (Invariant #9); HealthProjection rebuilds current health by replay, never re-probing (Invariant #8). Monitor is live-only, started in ServerModule.start alongside narration; seeded from the projection so a restart does not re-emit a degraded the log already records. First concrete probe: DiskWatermarkProbe — event-log + CAS size and growth rate, warning before the 2am surprise. Read surface mirrors `correx stats`: GET /health/checks replays the system session and `correx health` renders it. Health events ride the reserved __system__ session, filtered from operator session listings. LLAMA_SERVER (liveness + tokens/sec trend) and EVENT_STORE (append/fsync latency) probes plug into the same HealthProbe framework as follow-ups.
This commit is contained in:
@@ -16,6 +16,21 @@ data class CorrexConfig(
|
||||
val project: ProjectConfig = ProjectConfig(),
|
||||
val personalization: PersonalizationConfig = PersonalizationConfig(),
|
||||
val orchestration: OrchestrationKnobs = OrchestrationKnobs(),
|
||||
val health: HealthConfig = HealthConfig(),
|
||||
)
|
||||
|
||||
/**
|
||||
* Continuous health watch (observability-spec §4). When [enabled], a background monitor polls
|
||||
* the on-disk footprint every [intervalMs] and records a degraded/restored event on the edge.
|
||||
* [diskWarnBytes] is the absolute event-log + CAS size that trips a warning; [diskWarnGrowthBytesPerMin]
|
||||
* is the growth rate that warns log compaction is becoming non-theoretical. Defaults suit a homelab.
|
||||
*/
|
||||
@Serializable
|
||||
data class HealthConfig(
|
||||
val enabled: Boolean = true,
|
||||
val intervalMs: Long = 30_000,
|
||||
val diskWarnBytes: Long = 5_000_000_000,
|
||||
val diskWarnGrowthBytesPerMin: Long = 50_000_000,
|
||||
)
|
||||
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user