docs: add module documentation in AsciiDoc with PlantUML diagrams
Generated by per-group subagents covering all 52 modules across: core (18), infrastructure (10), apps (5), interfaces (3), plugins (7), testing (9) Documentation format: - AsciiDoc (.adoc) files in docs/modules/<group>/ - PlantUML (.puml) files in docs/diagrams/ - .adoc files reference diagrams via include:: directives Each doc covers: purpose, responsibilities, non-responsibilities, key types, event flow, integration points, invariants, PlantUML diagram, known issues, and open questions.
This commit is contained in:
@@ -0,0 +1,130 @@
|
||||
= infra-artifacts-cas
|
||||
|
||||
== purpose
|
||||
|
||||
Content-addressable storage (CAS) for binary artifact blobs, backed by append-only segment files on disk. Provides write-once, read-by-hash semantics with integrity verification (CRC32C + BLAKE3), automatic segment rotation, tail recovery after crash, compaction of dead data, and eviction of unreferenced segments. The index is maintained in a SQLite database.
|
||||
|
||||
== responsibilities
|
||||
|
||||
* Store binary blobs addressed by BLAKE3 hash (content-addressable)
|
||||
* Append records to immutable segment files with header (length, CRC32C, BLAKE3 hash)
|
||||
* Look up and read records by hash via a SQLite index
|
||||
* Detect and recover partially-written tail records after process crash
|
||||
* Compact fragmented segments: defragment live records into new segments, delete old ones
|
||||
* Evict dead segments (zero live bytes) when total storage exceeds `maxTotalBytes`
|
||||
* Materialize `FileWrittenPayload` to the filesystem for artifact writing (sandbox-aware)
|
||||
* Coordinate flush ordering with the event store to maintain consistency
|
||||
|
||||
== non-responsibilities
|
||||
|
||||
* Does not manage artifact lifecycle (creation, validation, archival) — that belongs to `core:artifacts`
|
||||
* Does not store event data — that belongs to `infra:persistence`
|
||||
* Does not enforce access control — that belongs to `infra:security`
|
||||
* Does not provide a queryable artifact index beyond hash lookup
|
||||
|
||||
== key types
|
||||
|
||||
=== CasArtifactStore
|
||||
* **kind**: class
|
||||
* **purpose**: Main entry point implementing `ArtifactStore`. Manages a `SegmentWriter` and `SegmentReader`, coordinates compaction and eviction with mutexes, and provides crash recovery via `TailScanner`.
|
||||
* **fields**: `config`, `index`, `segmentsDir`, `writer`, `reader`
|
||||
|
||||
=== CasConfig
|
||||
* **kind**: data class
|
||||
* **purpose**: Configuration for the CAS store.
|
||||
* **fields**: `rootDir` (filesystem root), `maxSegmentBytes` (default 64MB), `maxTotalBytes` (default `Long.MAX_VALUE`)
|
||||
|
||||
=== ArtifactIndex (interface)
|
||||
* **kind**: interface
|
||||
* **purpose**: Contract for the hash-to-location index. Supports lookup, insert, update, and segment-scoped iteration/deletion.
|
||||
|
||||
=== SqliteArtifactIndex
|
||||
* **kind**: class
|
||||
* **purpose**: SQLite-backed implementation of `ArtifactIndex`. Uses WAL mode and `INSERT OR IGNORE` / `ON CONFLICT DO UPDATE` semantics.
|
||||
|
||||
=== SegmentLayout
|
||||
* **kind**: object
|
||||
* **purpose**: Binary layout constants and encoding/decoding utilities. Record header = 4 bytes payload length + 4 bytes CRC32C + 32 bytes BLAKE3 hash.
|
||||
* **constants**: `HASH_SIZE = 32`, `HEADER_SIZE = 40`
|
||||
|
||||
=== SegmentWriter
|
||||
* **kind**: class
|
||||
* **purpose**: Writes records to an append-only segment file. Rotates to a new file when the current segment exceeds `maxSegmentBytes`. Supports `fsync()`.
|
||||
|
||||
=== SegmentReader
|
||||
* **kind**: class
|
||||
* **purpose**: Reads a record from a segment file by `Location`, verifying CRC32C and BLAKE3 hash on read. Throws `CorruptRecordException` on mismatch.
|
||||
|
||||
=== Location
|
||||
* **kind**: data class
|
||||
* **purpose**: Pointer to a record within a segment file.
|
||||
* **fields**: `segmentId`, `offset`, `length`
|
||||
|
||||
=== TailScanner
|
||||
* **kind**: class
|
||||
* **purpose**: After a crash, scans the active segment from the last known tail offset, recovers intact records, and truncates the file at the first corrupted byte.
|
||||
* **fields**: `segmentsDir`, `index`
|
||||
|
||||
=== LivenessScanner
|
||||
* **kind**: class
|
||||
* **purpose**: Scans the event store for all referenced artifact IDs (`InferenceStartedEvent.promptArtifactId`, `InferenceCompletedEvent.responseArtifactId`) and computes per-segment live-byte statistics.
|
||||
|
||||
=== Compactor
|
||||
* **kind**: class
|
||||
* **purpose**: Selects segments below a live-ratio threshold (default 0.5), reads live entries, writes them to a new segment, atomically swaps index entries, and deletes old segment files.
|
||||
* **fields**: `config`, `index`, `liveness`, `storeMutex`
|
||||
|
||||
=== Evictor
|
||||
* **kind**: class
|
||||
* **purpose**: When total storage exceeds `maxTotalBytes`, deletes segments that have zero live bytes (not the active segment).
|
||||
* **fields**: `config`, `index`, `liveness`
|
||||
|
||||
=== DefaultMaterializingArtifactWriter
|
||||
* **kind**: class
|
||||
* **purpose**: Implements `MaterializingArtifactWriter` by writing `FileWrittenPayload` to the filesystem under a sandbox root, computing the BLAKE3 hash, and returning a `FileWrittenArtifact`. Prevents path traversal.
|
||||
|
||||
== event flow
|
||||
|
||||
*Inbound:*
|
||||
* `InferenceStartedEvent` / `InferenceCompletedEvent` — consumed by `LivenessScanner` to determine which artifacts are live (referenced by the event log)
|
||||
|
||||
*Outbound:* None. The CAS store does not emit events.
|
||||
|
||||
== integration points
|
||||
|
||||
* `core:artifactstore` — `ArtifactStore` (interface implemented by `CasArtifactStore`)
|
||||
* `core:artifacts` — `MaterializingArtifactWriter`, `MaterializationResult`, `FileWrittenArtifact`, `FileWrittenPayload`
|
||||
* `core:events` — `StoredEvent`, `InferenceStartedEvent`, `InferenceCompletedEvent`, `ArtifactId`
|
||||
* `infra:persistence` — `EventStore` (consumed by `LivenessScanner`)
|
||||
|
||||
== invariants
|
||||
|
||||
* Records are immutable once written — updates are not possible (new hash = new record)
|
||||
* Each record is verified on read: CRC32C and BLAKE3 must match the header
|
||||
* The `Compactor` holds `storeMutex` during the atomic swap to prevent concurrent puts from observing stale locations
|
||||
* Compaction and eviction are mutually exclusive (`maintenanceMutex`)
|
||||
* `TailScanner` truncates the active segment at the first corrupt record before the writer resumes
|
||||
* The `flushBefore` protocol coordinates event store and CAS flushing to maintain write ordering
|
||||
|
||||
== PlantUML diagram
|
||||
|
||||
[plantuml, infra-artifacts-cas, "png"]
|
||||
----
|
||||
include::../../diagrams/infra-artifacts-cas.puml[]
|
||||
----
|
||||
|
||||
|
||||
|
||||
|
||||
== known issues
|
||||
|
||||
* `LivenessScanner.collectLiveIds()` iterates all events in the store — O(n) per compaction/eviction cycle, may be slow for large event logs
|
||||
* `Evictor` only evicts segments with zero live bytes — partially-dead segments are left for compaction
|
||||
* `DefaultMaterializingArtifactWriter.materialize()` computes BLAKE3 from a fresh byte array read after writing — could compute during write instead
|
||||
* No garbage collection for the SQLite index — deleted segment entries leave behind index rows that are only removed via `deleteEntriesIn`
|
||||
|
||||
== open questions
|
||||
|
||||
* Should liveness scanning support incremental updates via event subscription rather than full replay?
|
||||
* Should segment files be encrypted at rest?
|
||||
* Should there be a maximum segment count to bound file descriptor usage?
|
||||
Reference in New Issue
Block a user