feat(server,cli): event-sourced health checks — disk watermark probe (observability §4)

Health states are events. A HealthMonitor polls registered HealthProbes on an
interval and appends HealthDegraded/HealthRestored only on a status transition
(hysteresis), so health is observable, replayable, and renderable like every
other fact in the log. The probe reads nondeterministic environment state and
records it at observe-time (Invariant #9); HealthProjection rebuilds current
health by replay, never re-probing (Invariant #8). Monitor is live-only, started
in ServerModule.start alongside narration; seeded from the projection so a
restart does not re-emit a degraded the log already records.

First concrete probe: DiskWatermarkProbe — event-log + CAS size and growth rate,
warning before the 2am surprise. Read surface mirrors `correx stats`:
GET /health/checks replays the system session and `correx health` renders it.
Health events ride the reserved __system__ session, filtered from operator
session listings.

LLAMA_SERVER (liveness + tokens/sec trend) and EVENT_STORE (append/fsync latency)
probes plug into the same HealthProbe framework as follow-ups.
This commit is contained in:
2026-06-13 22:19:33 +04:00
parent 3467826d4a
commit f8fd2601a8
19 changed files with 949 additions and 3 deletions
@@ -16,6 +16,21 @@ data class CorrexConfig(
val project: ProjectConfig = ProjectConfig(),
val personalization: PersonalizationConfig = PersonalizationConfig(),
val orchestration: OrchestrationKnobs = OrchestrationKnobs(),
val health: HealthConfig = HealthConfig(),
)
/**
* Continuous health watch (observability-spec §4). When [enabled], a background monitor polls
* the on-disk footprint every [intervalMs] and records a degraded/restored event on the edge.
* [diskWarnBytes] is the absolute event-log + CAS size that trips a warning; [diskWarnGrowthBytesPerMin]
* is the growth rate that warns log compaction is becoming non-theoretical. Defaults suit a homelab.
*/
@Serializable
data class HealthConfig(
val enabled: Boolean = true,
val intervalMs: Long = 30_000,
val diskWarnBytes: Long = 5_000_000_000,
val diskWarnGrowthBytesPerMin: Long = 50_000_000,
)
/**