kami
9734eec63c
Generated by per-group subagents covering all 52 modules across: core (18), infrastructure (10), apps (5), interfaces (3), plugins (7), testing (9) Documentation format: - AsciiDoc (.adoc) files in docs/modules/<group>/ - PlantUML (.puml) files in docs/diagrams/ - .adoc files reference diagrams via include:: directives Each doc covers: purpose, responsibilities, non-responsibilities, key types, event flow, integration points, invariants, PlantUML diagram, known issues, and open questions.
131 lines
6.3 KiB
Plaintext
131 lines
6.3 KiB
Plaintext
= infra-artifacts-cas
|
|
|
|
== purpose
|
|
|
|
Content-addressable storage (CAS) for binary artifact blobs, backed by append-only segment files on disk. Provides write-once, read-by-hash semantics with integrity verification (CRC32C + BLAKE3), automatic segment rotation, tail recovery after crash, compaction of dead data, and eviction of unreferenced segments. The index is maintained in a SQLite database.
|
|
|
|
== responsibilities
|
|
|
|
* Store binary blobs addressed by BLAKE3 hash (content-addressable)
|
|
* Append records to immutable segment files with header (length, CRC32C, BLAKE3 hash)
|
|
* Look up and read records by hash via a SQLite index
|
|
* Detect and recover partially-written tail records after process crash
|
|
* Compact fragmented segments: defragment live records into new segments, delete old ones
|
|
* Evict dead segments (zero live bytes) when total storage exceeds `maxTotalBytes`
|
|
* Materialize `FileWrittenPayload` to the filesystem for artifact writing (sandbox-aware)
|
|
* Coordinate flush ordering with the event store to maintain consistency
|
|
|
|
== non-responsibilities
|
|
|
|
* Does not manage artifact lifecycle (creation, validation, archival) — that belongs to `core:artifacts`
|
|
* Does not store event data — that belongs to `infra:persistence`
|
|
* Does not enforce access control — that belongs to `infra:security`
|
|
* Does not provide a queryable artifact index beyond hash lookup
|
|
|
|
== key types
|
|
|
|
=== CasArtifactStore
|
|
* **kind**: class
|
|
* **purpose**: Main entry point implementing `ArtifactStore`. Manages a `SegmentWriter` and `SegmentReader`, coordinates compaction and eviction with mutexes, and provides crash recovery via `TailScanner`.
|
|
* **fields**: `config`, `index`, `segmentsDir`, `writer`, `reader`
|
|
|
|
=== CasConfig
|
|
* **kind**: data class
|
|
* **purpose**: Configuration for the CAS store.
|
|
* **fields**: `rootDir` (filesystem root), `maxSegmentBytes` (default 64MB), `maxTotalBytes` (default `Long.MAX_VALUE`)
|
|
|
|
=== ArtifactIndex (interface)
|
|
* **kind**: interface
|
|
* **purpose**: Contract for the hash-to-location index. Supports lookup, insert, update, and segment-scoped iteration/deletion.
|
|
|
|
=== SqliteArtifactIndex
|
|
* **kind**: class
|
|
* **purpose**: SQLite-backed implementation of `ArtifactIndex`. Uses WAL mode and `INSERT OR IGNORE` / `ON CONFLICT DO UPDATE` semantics.
|
|
|
|
=== SegmentLayout
|
|
* **kind**: object
|
|
* **purpose**: Binary layout constants and encoding/decoding utilities. Record header = 4 bytes payload length + 4 bytes CRC32C + 32 bytes BLAKE3 hash.
|
|
* **constants**: `HASH_SIZE = 32`, `HEADER_SIZE = 40`
|
|
|
|
=== SegmentWriter
|
|
* **kind**: class
|
|
* **purpose**: Writes records to an append-only segment file. Rotates to a new file when the current segment exceeds `maxSegmentBytes`. Supports `fsync()`.
|
|
|
|
=== SegmentReader
|
|
* **kind**: class
|
|
* **purpose**: Reads a record from a segment file by `Location`, verifying CRC32C and BLAKE3 hash on read. Throws `CorruptRecordException` on mismatch.
|
|
|
|
=== Location
|
|
* **kind**: data class
|
|
* **purpose**: Pointer to a record within a segment file.
|
|
* **fields**: `segmentId`, `offset`, `length`
|
|
|
|
=== TailScanner
|
|
* **kind**: class
|
|
* **purpose**: After a crash, scans the active segment from the last known tail offset, recovers intact records, and truncates the file at the first corrupted byte.
|
|
* **fields**: `segmentsDir`, `index`
|
|
|
|
=== LivenessScanner
|
|
* **kind**: class
|
|
* **purpose**: Scans the event store for all referenced artifact IDs (`InferenceStartedEvent.promptArtifactId`, `InferenceCompletedEvent.responseArtifactId`) and computes per-segment live-byte statistics.
|
|
|
|
=== Compactor
|
|
* **kind**: class
|
|
* **purpose**: Selects segments below a live-ratio threshold (default 0.5), reads live entries, writes them to a new segment, atomically swaps index entries, and deletes old segment files.
|
|
* **fields**: `config`, `index`, `liveness`, `storeMutex`
|
|
|
|
=== Evictor
|
|
* **kind**: class
|
|
* **purpose**: When total storage exceeds `maxTotalBytes`, deletes segments that have zero live bytes (not the active segment).
|
|
* **fields**: `config`, `index`, `liveness`
|
|
|
|
=== DefaultMaterializingArtifactWriter
|
|
* **kind**: class
|
|
* **purpose**: Implements `MaterializingArtifactWriter` by writing `FileWrittenPayload` to the filesystem under a sandbox root, computing the BLAKE3 hash, and returning a `FileWrittenArtifact`. Prevents path traversal.
|
|
|
|
== event flow
|
|
|
|
*Inbound:*
|
|
* `InferenceStartedEvent` / `InferenceCompletedEvent` — consumed by `LivenessScanner` to determine which artifacts are live (referenced by the event log)
|
|
|
|
*Outbound:* None. The CAS store does not emit events.
|
|
|
|
== integration points
|
|
|
|
* `core:artifactstore` — `ArtifactStore` (interface implemented by `CasArtifactStore`)
|
|
* `core:artifacts` — `MaterializingArtifactWriter`, `MaterializationResult`, `FileWrittenArtifact`, `FileWrittenPayload`
|
|
* `core:events` — `StoredEvent`, `InferenceStartedEvent`, `InferenceCompletedEvent`, `ArtifactId`
|
|
* `infra:persistence` — `EventStore` (consumed by `LivenessScanner`)
|
|
|
|
== invariants
|
|
|
|
* Records are immutable once written — updates are not possible (new hash = new record)
|
|
* Each record is verified on read: CRC32C and BLAKE3 must match the header
|
|
* The `Compactor` holds `storeMutex` during the atomic swap to prevent concurrent puts from observing stale locations
|
|
* Compaction and eviction are mutually exclusive (`maintenanceMutex`)
|
|
* `TailScanner` truncates the active segment at the first corrupt record before the writer resumes
|
|
* The `flushBefore` protocol coordinates event store and CAS flushing to maintain write ordering
|
|
|
|
== PlantUML diagram
|
|
|
|
[plantuml, infra-artifacts-cas, "png"]
|
|
----
|
|
include::../../diagrams/infra-artifacts-cas.puml[]
|
|
----
|
|
|
|
|
|
|
|
|
|
== known issues
|
|
|
|
* `LivenessScanner.collectLiveIds()` iterates all events in the store — O(n) per compaction/eviction cycle, may be slow for large event logs
|
|
* `Evictor` only evicts segments with zero live bytes — partially-dead segments are left for compaction
|
|
* `DefaultMaterializingArtifactWriter.materialize()` computes BLAKE3 from a fresh byte array read after writing — could compute during write instead
|
|
* No garbage collection for the SQLite index — deleted segment entries leave behind index rows that are only removed via `deleteEntriesIn`
|
|
|
|
== open questions
|
|
|
|
* Should liveness scanning support incremental updates via event subscription rather than full replay?
|
|
* Should segment files be encrypted at rest?
|
|
* Should there be a maximum segment count to bound file descriptor usage?
|