fix(server,persistence): stop slow WS client stalling kernel; log dropped emits (#53)
Two event-delivery back-pressure faults from the core audit (#7). (a) streamGlobal forwarded globalFlow → a per-connection Channel with buffer.send (SUSPEND). globalFlow is also SUSPEND-overflow, so one wedged WebSocket client back-pressured globalFlow → append() suspended → the whole kernel stalled. Now the collector uses trySend; on overflow it closes the buffer with SlowClientException, tearing down that one connection (client reconnects and re-syncs via replaySnapshot) instead of blocking append(). (b) Per-session subscriptions (extraBufferCapacity 64) used tryEmit with the return ignored: a lagging in-process collector (e.g. LiveArtifactRepository) silently diverged from the durable log. Both SqliteEventStore and InMemoryEventStore now warnIfDropped() — the event is still persisted; only the live SharedFlow drops, and we surface it. ponytail: log-only; bounded rebuild-on-lag if divergence ever bites. Added slf4j-api to :infrastructure:persistence (already the standard logging dep across modules) for the warn. Green: :infrastructure:persistence + :apps:server compile+detekt+test (the one pre-existing ModelLifecycleWiringTest failure needs a real llama-server, fails identically on clean checkout).
This commit is contained in:
@@ -65,6 +65,9 @@ import java.util.UUID
|
||||
|
||||
private val log = LoggerFactory.getLogger(GlobalStreamHandler::class.java)
|
||||
private const val BUFFER_CAPACITY = 1024
|
||||
|
||||
/** Raised to tear down a WS connection whose forward buffer overflowed, so the client reconnects. */
|
||||
private class SlowClientException : RuntimeException("global WS stream buffer overflow")
|
||||
private const val RESOURCE_PUSH_INTERVAL_MS = 2500L
|
||||
private const val BYTES_PER_MB = 1024L * 1024L
|
||||
|
||||
@@ -788,8 +791,17 @@ internal suspend fun streamGlobal(
|
||||
is SharedFlow<StoredEvent> -> source.onSubscription { subscribed.complete(Unit) }
|
||||
else -> source.onStart { subscribed.complete(Unit) }
|
||||
}
|
||||
// trySend, never send: a blocking send here would back-pressure globalFlow (SUSPEND overflow),
|
||||
// suspending append() and stalling the whole kernel on one wedged client. On overflow we instead
|
||||
// drop the connection (close the buffer); the client reconnects and re-syncs via replaySnapshot.
|
||||
val subscription = launch {
|
||||
signaled.collect { buffer.send(it) }
|
||||
signaled.collect {
|
||||
val result = buffer.trySend(it)
|
||||
if (result.isFailure && !result.isClosed) {
|
||||
log.warn("global WS stream buffer overflow (slow client); dropping connection to re-sync")
|
||||
buffer.close(SlowClientException())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
|
||||
Reference in New Issue
Block a user