feat: rehydrate L3 metadata from event log on startup

TurboVec persists vectors via the sidecar but kept entry metadata
(sessionId/turnId/text) only in an in-memory map, so restarting with
backend=turbovec silently dropped every query hit (the id→metadata
lookup missed). Rebuild that map from the ChatTurnEvent log at startup.

- RehydratableL3MemoryStore: capability interface for stores whose
  vectors persist independently of the JVM. TurboVec implements it;
  InMemory deliberately does not (its vectors are volatile too).
- L3MetadataRehydrator replays ChatTurnEvents into the metadata map.
  No embedder: the query path never reads entry.vector, so vectors
  stay in the sidecar and metadata comes from the event log (the
  source of truth, invariant #1). Short-circuits for non-rehydratable
  backends before scanning the log.
- Wired into Main before the server accepts queries; no sidecar spawn.

Backfill of never-embedded history (needs the embedder) is a separate
follow-up. The in_memory non-durability warning already exists.
This commit is contained in:
2026-05-31 22:08:52 +04:00
parent d03479cea7
commit 5beb866036
5 changed files with 181 additions and 3 deletions
@@ -0,0 +1,40 @@
package com.correx.core.router.l3
import com.correx.core.events.events.ChatTurnEvent
import com.correx.core.events.stores.EventStore
import org.slf4j.LoggerFactory
/**
* Rebuilds a [RehydratableL3MemoryStore]'s in-memory metadata map from the
* `ChatTurnEvent` log at startup.
*
* Persisted vectors (e.g. the TurboVec sidecar index) survive a restart, but the
* metadata map does not; without this every L3 query silently drops its hits
* because the `id → metadata` lookup misses. Rehydration reads only the event log
* (no embedder), so it is cheap and replay-safe. Stores without independently
* persisted vectors are not rehydratable and are skipped.
*/
class L3MetadataRehydrator(private val eventStore: EventStore) {
private val log = LoggerFactory.getLogger(L3MetadataRehydrator::class.java)
/** @return the number of entries rehydrated (0 if [store] is not rehydratable). */
suspend fun rehydrate(store: L3MemoryStore): Int {
if (store !is RehydratableL3MemoryStore) return 0
val entries = eventStore.allEvents()
.mapNotNull { it.payload as? ChatTurnEvent }
.map { turn ->
L3MemoryEntry(
id = turn.turnId,
sessionId = turn.sessionId,
turnId = turn.turnId,
text = turn.content,
vector = FloatArray(0),
timestampMs = turn.timestampMs,
)
}
.toList()
store.rehydrateMetadata(entries)
log.info("L3 metadata rehydrated: {} chat turns from event log", entries.size)
return entries.size
}
}
@@ -0,0 +1,23 @@
package com.correx.core.router.l3
/**
* An [L3MemoryStore] whose vectors persist independently of the JVM (e.g. a
* sidecar index on disk) but whose entry metadata (`sessionId`/`turnId`/`text`)
* lives in memory and is therefore lost on restart.
*
* Such a store can rebuild that metadata from the `ChatTurnEvent` log at startup:
* the event log is the source of truth (invariant #1) and the metadata map is a
* disposable projection. Only the metadata map is loaded — the persisted vectors
* are never re-sent, so rehydration needs no embedder.
*
* Stores whose vectors are themselves volatile (e.g. [InMemoryL3MemoryStore])
* deliberately do NOT implement this: rehydrating metadata without matching
* vectors would be meaningless.
*/
interface RehydratableL3MemoryStore : L3MemoryStore {
/**
* Load entry metadata into the in-memory map without re-sending vectors.
* Idempotent: re-running must not clobber entries already present.
*/
suspend fun rehydrateMetadata(entries: List<L3MemoryEntry>)
}