Commit Graph

39 Commits

Author SHA1 Message Date
kami 047e2a4070 feat(context,infra): compression pipeline stages 4-5 — token pruning + relevance + ToMe
- build() is now suspend: pipeline runs all stages in fixed order, each gated by level
- TOKEN_PRUNE (level 3): TokenPruner interface + LLMLingua-2 sidecar (sidecars/llmlingua)
  + HttpTokenPruner adapter (fails open if sidecar down); prunes freeform, preserves
  protected spans, skips tier-0 turns when TIER_SPLIT on
- TOME_MERGE (level 8): ToMeMerger collapses near-duplicate freeform turns (Jaccard)
- Stage 5 selection: RelevanceScorer + EmbeddingRelevanceScorer (cosine over Embedder);
  query-conditioned reorder so least-relevant freeform drops first under budget
- [orchestration] compression_level + token_pruner_url config, wired in Main
- suspend ripple fixed across builder callers/stubs
2026-07-01 14:29:56 +04:00
kami 61be90070f feat(context): compression pipeline stage 1 — level policy + deterministic stages
Additive CompressionPolicy (levels 1-9), ProtectedSpanTagger (regex
load-bearing spans), FormatCompressor (json→dotted lines), ContextClassifier
(static/structured/freeform). Stage 1 (FORMAT_COMPRESS) wired into
DefaultContextPackBuilder behind the policy; structured entries only, lossless.
PromptRenderer repetition-anchoring of steering directives.
2026-07-01 13:53:19 +04:00
kami d26f20c316 docs(dox): build out the DOX AGENTS.md hierarchy
Root Child DOX Index assembled, plus a per-module AGENTS.md across the tree
(core/*, infrastructure/*, apps/*, testing/*, and docs/examples/frontend/etc),
each following the DOX section shape: Purpose, Ownership, Local Contracts,
Work Guidance, Verification, Child DOX Index.
2026-06-28 20:43:27 +04:00
kami 400ae5729a feat(router): rubber-duck idea capture + feed
The rubber-duck path emits a fenced {"ideas":[…]} block; Ideas.parse strips it
from the chat turn and onUserInput records an IdeaCapturedEvent each (CHAT only).
IdeaReader folds eventStore.allEvents() into the active cross-session board, and
DefaultRouterContextBuilder injects a capped "## Ideas" entry — router-only, so
workflow stages never see it. Extended SYSTEM_PROMPT option (d) to capture ideas.
2026-06-14 14:25:49 +04:00
kami fc373e11eb feat(router): emit structured workflow proposals from triage
The triage system prompt now appends a fenced json propose_workflows block;
WorkflowProposals.parse strips it from the visible chat turn and onUserInput
records a WorkflowProposedEvent (CHAT only). Candidate ids are validated against
the registered workflows (invariant #7) so an invented id never reaches the
operator. Bumped a token-budget test for the larger prompt.
2026-06-14 13:28:51 +04:00
kami 35e921c6d1 fix(inference): tokenize endpoint sent wrong field — every token estimate was 0
LlamaCppTokenizer posted {"tokens":[text]} but llama.cpp's /tokenize expects
{"content":text}; the server silently answered {"tokens":[]}, so countTokens
returned 0 for every string. Every context entry carried tokenEstimate=0 and
the whole budget/trim pipeline was blind — live QA saw a 34k-token prompt
sail through a 16384 stage budget (three raw file_read results of plan docs),
crater t/s, degrade output, and fail the stage on validation 3x.

Also guard estimateTokens: a zero count for non-blank content falls back to
the chars/4 heuristic so a lying tokenizer can never blind budgeting again.
2026-06-11 22:50:53 +04:00
kami 07819ec82d fix(inference): fold non-L0 SYSTEM entries into the leading system message
Strict chat templates (Qwen3.5) reject any system message after index 0.
Recalled L3 memory, L2 stage summaries, and retrieval entries carry
EntryRole.SYSTEM and rendered as standalone mid-conversation system turns,
making llama-server 400 on every router turn once L3 recall returned hits.
2026-06-11 21:55:28 +04:00
kami 9d38a89c88 feat(memory): state-keyed repo-map reuse via L3 presence check; embed entries on fresh scan
- Add existsByTurnIdPrefix(prefix) to L3MemoryStore interface; implemented in
  InMemoryL3MemoryStore (mutex-guarded) and TurboVecL3MemoryStore (metadata map).
  All test fakes (TrackingL3MemoryStore, CapturingStore) updated.
- Introduce RepoMapIndexerPort interface; RepoMapIndexer implements it. Allows
  injection of test doubles without framework overhead.
- ProjectMemoryService.indexAndRecord: reads latest WorkspaceStateObservedEvent
  from event store (invariant #9 — recorded fact, not re-probe); skips scan+embed
  when L3 already holds entries for "repomap:<root>:<stateKey>"; passes stateKey
  to RepoMapComputedEvent; embeds each scanned entry after append; per-entry
  failures warn-logged (CancellationException rethrown).
- Tests: InMemoryL3MemoryStoreTest +2 cases; new ProjectMemoryServiceReuseTest
  covers same-key skip, changed-key re-index, no-observation always-reindex,
  and embedding turnId tag verification.
2026-06-11 13:30:57 +04:00
kami f1fc43cc85 feat(router,server): wire workflow inventory and project profile into router context per turn 2026-06-11 13:15:32 +04:00
kami 72e7002623 feat(router): replace bare refusal with triage directive in system prompt 2026-06-11 11:15:59 +04:00
kami a32e9acac4 feat(router): inject available-workflows and project-profile into router L0 context 2026-06-11 11:11:04 +04:00
kami 883e23dec7 feat(router,inference,config): pin narration to a configured model_id
[router.narration] model_id selects which provider handles narration turns
instead of generic capability routing — lets a small/fast model own narration
while the main model keeps CHAT/STEERING.

- NarrationSettings.modelId parsed from TOML, threaded via RouterConfig
  .narrationModelId into DefaultRouterFacade.narrate (3-arg route)
- DefaultInferenceRouter: exact-id match against healthy providers, with a
  post-select health check; any miss logs WARN and falls back to capability
  routing (never fails the narration turn)
- onUserInput unchanged — only narrate uses the pinned model (tested)
2026-06-10 20:53:28 +04:00
kami e6084dcbda feat(router): proper grounded system prompts for chat and narration
Replace the placeholder "You are a routing assistant" one-liner with two
purpose-written prompts: a conversational prompt that frames the router as
correx's operator-facing interface to the event-sourced engine (grounded in
workflow state, no fabrication, concise, steering-aware), and a dedicated
narrator prompt for buildNarrationContext that asks for one or two present-tense
sentences naming the stage and outcome.

The longer protected frame shifted token budgets, so the two L2 eviction tests
now size their budget from the measured protected-frame + per-entry cost instead
of hard-coded magic numbers, making them robust to prompt length.
2026-06-03 22:44:32 +04:00
kami ad3ec1965d fix(router,kernel): narration user turn + stage transition ordering
Two issues surfaced during live QA once workflows began progressing past the
first stage:

1. Narration prompts went out with no user message, so the chat template
   rejected them ("No user query found in messages"). buildNarrationContext
   filed the trigger instruction (a user turn) under ContextLayer.L0, and
   PromptRenderer folds every L0 entry into the system message regardless of
   role. Move the trigger to L1 so it renders as the user turn.

2. TransitionExecutedEvent was emitted after the next stage had already run:
   executeMove called enterStage (which executes the stage) before advanceStage
   (which emits the transition), so the event log showed a stage running before
   the event marking entry into it. Emit the transition before entering.

Adds a regression test asserting a rendered narration prompt carries a user
message.
2026-06-03 22:38:12 +04:00
kami 8fe3a504bb fix(context): preserve chronological turn order through pack assembly
Tool-calling loops were rendered as all-assistant-calls-then-all-tool-results
with the original task last, causing the model to re-issue the same tool call
indefinitely. Two stacked reorderings caused it:

- DefaultContextPackBuilder grouped entries by sourceType for per-type
  compression, discarding a,t,a,t interleaving.
- PromptRenderer forced L1 (the live user turn) to render last, pushing the
  task after the entire transcript.
- SessionOrchestrator's tool loop re-fed currentContext.layers.values.flatten()
  (grouped by layer) each round, compounding the scramble.

Add a chronological `ordinal` to ContextEntry, stamped by the builder from
input order and restored after grouping/compression (the compressor preserves
entry identity, so ordinals survive). PromptRenderer now orders non-system
messages by ordinal, with the old L1-last layer priority kept only as a
tiebreak so router chat (ordinal 0) is unchanged. The orchestrator keeps a
running ordered accumulator instead of reading back the grouped pack.

Adds builder + renderer ordering regression tests.
2026-06-03 22:22:52 +04:00
kami 38b007a6cd feat(router): Task 3.2 — NarrationTrigger + buildNarrationContext + RouterFacade.narrate
- Add NarrationTrigger(kind, instruction) model to core:router
- Add RouterContextBuilder.buildNarrationContext: minimal ContextPack
  (system + workflow status + L2 + trigger instruction); excludes
  conversationHistory and L3 recalled memory
- Add RouterFacade.narrate: emits RouterNarrationEvent with content,
  latencyMs, tokensUsed; skips event on blank inference; never writes
  ChatTurnEvent or l3MemoryStore.store
- Add RouterNarrationTest covering all acceptance criteria
- Add narrate default to RouterContextBuilder interface so existing
  anonymous test stubs compile without a stub override
2026-06-03 15:52:08 +04:00
kami 8d8b2914e0 feat(router): capture latency + tokens on ROUTER ChatTurnEvent
ChatTurnEvent gains nullable latencyMs and tokensUsed fields (defaults preserve
backward-compat; legacy JSON without them deserializes cleanly). emitChatTurn
accepts optional metrics; the ROUTER emit site passes inferenceResponse values
through; the USER emit site leaves both null.
2026-06-03 15:11:37 +04:00
kami 1dc7eae8a1 feat(router): enrich L2 decision points with steering and failure reason
Fix broken stage-summary interpolation ($payload.stageId rendered the event's toString()). RouterL2Entry gains reason + steeringNotes; the reducer folds SteeringNoteAddedEvent into pendingSteeringNotes and harvests them into the completing stage's L2 entry, then clears (adr-0003 §4, pure field extraction, zero inference). RouterContextBuilder renders summary + reason + steering.
2026-06-02 13:58:41 +04:00
kami 5beb866036 feat: rehydrate L3 metadata from event log on startup
TurboVec persists vectors via the sidecar but kept entry metadata
(sessionId/turnId/text) only in an in-memory map, so restarting with
backend=turbovec silently dropped every query hit (the id→metadata
lookup missed). Rebuild that map from the ChatTurnEvent log at startup.

- RehydratableL3MemoryStore: capability interface for stores whose
  vectors persist independently of the JVM. TurboVec implements it;
  InMemory deliberately does not (its vectors are volatile too).
- L3MetadataRehydrator replays ChatTurnEvents into the metadata map.
  No embedder: the query path never reads entry.vector, so vectors
  stay in the sidecar and metadata comes from the event log (the
  source of truth, invariant #1). Short-circuits for non-rehydratable
  backends before scanning the log.
- Wired into Main before the server accepts queries; no sidecar spawn.

Backfill of never-embedded history (needs the embedder) is a separate
follow-up. The in_memory non-durability warning already exists.
2026-05-31 22:08:52 +04:00
kami 8e6a3e1470 feat: inject recalled L3 memory into router context with budget
Record L3 retrieval as an event carrying hit text (invariant #9), then
rebuild router state and inject recalled memories as a SYSTEM L3 layer in
the context pack. Apply token budget: protected frames (L0 immutable +
current user turn) are never dropped; honest budgetUsed is reported and
'BudgetExceeded' is flagged in appliedStrategies when they overflow. L1/L2
fit newest-first so oldest entries drop first. Emit ContextTruncatedEvent
when entries are dropped. L3MemoryRetrievedEvent is emitted on every CHAT
turn (empty hits reset recalled memory).
2026-05-30 18:40:08 +04:00
kami e6239515bd feat: L3 memory retrieval on the router read path (record-only)
When a user sends input, embed it, query L3 across all sessions, dedup
against in-session turns, and record the retrieval as an event so replay
is deterministic (invariant #9). Hits land in RouterState; context
injection follows in a later slice.

- Add L3MemoryRetrievedEvent + L3RetrievedHit (registered in eventModule)
- RouterState.lastRetrievedMemory + reducer case; RouterTurn carries turnId
- RouterConfig.retrievalK (default 5)
- Harden L3 write path: runCatching + visible logging, cancellation
  re-thrown; event append stays ahead of the L3 write
- Warn prominently when the non-durable in_memory L3 backend is selected
2026-05-30 14:36:30 +04:00
kami 9f171d3236 feat: wire L3 write path via Embedder into router
Introduces Embedder contract in core:inference with a NoopEmbedder
default (zero vector) and wires both Embedder and L3MemoryStore into
DefaultRouterFacade. Each ChatTurnEvent emission now embeds the turn
content and writes a corresponding L3MemoryEntry, so cross-session
memory is captured at the source of truth (the event emission point).

InfrastructureModule defaults to NoopEmbedder + InMemoryL3MemoryStore
so the system runs without an external embedding model or vector store
wired. The TurboVec adapter stays unwired for now; switching to it is
a configuration concern (Epic 12 config layer).

Also gitignores apps/tui/logs/ (Kotlin TUI runtime logs, pre-Go rewrite).
2026-05-30 01:01:40 +04:00
kami a3f29e6eb9 feat: event-source router CHAT and STEERING conversation turns
Adds ChatTurnEvent (role USER|ROUTER) so router conversation history
survives restart and is rebuilt from the event log via projection,
restoring Hard Invariant #1 for the CHAT path. Both CHAT and STEERING
now emit user + router turn events; STEERING additionally keeps its
existing SteeringNoteAddedEvent so the structured directive record is
preserved alongside the conversation turns.

RouterFacade drops the in-memory ConcurrentHashMap of histories and
rebuilds state through the repository after each emission. Reducer
appends turns to conversationHistory. ChatTurnEvent is registered in
eventModule.
2026-05-30 00:53:26 +04:00
kami 3981d8443d fix: complete P4 audit findings — steering wiring, generation config
P4-1: Wire steering to actually work (advisory-only):
  - RouterFacade emits SteeringNoteAddedEvent via event store in STEERING mode
  - SessionOrchestrator reads steering notes from event store and includes them
    as ContextEntry objects (sourceType=steeringNote) in stage context packs
  - Delete dead SteeringNote data class; single vocabulary = SteeringNoteAddedEvent
  - Add optional validateSteering callback to DefaultRouterFacade for future
    ValidationPipeline wiring (avoids cross-core dependency)
  - Update RouterFacadeTest to assert event emission

P4-2: Router FACE — move hardcoded generation config:
  - Add GenerationConfig field to RouterConfig with same defaults
  - RouterFacade uses config.generationConfig instead of inline literals
  - (Remaining P4-2 items: conversation persistence and coordination
    semantics are design decisions requiring ADRs, deferred)
2026-05-29 01:19:21 +04:00
kami 7936251d6b fix: complete all P2 audit findings — dead code, NPE guard, hygiene
P2-1: Guard SessionId NPE with safe-call in SessionsReducer
P2-2: Remove 16 dead event classes + reducer branches; replace with live orchestration events in tests
P2-3: Standardize ChatInput divergences — guard CancelSession, single-arg ProtocolError
P2-4: Remove dead TuiToolRecord.diff and ToolDisplayStatus.REQUESTED
P2-5: Remove dead streamLive from SessionEventBridge
P2-6: Extract SessionsReducerContext data class for 6+ param method
P2-7: Replace StageId("none") sentinel with TypeId.NONE
P2-9: Add TypeId.random() factory on all type-alias IDs
P2-10: Add KDoc on schemaVersion documenting reserved-for-migration
P2-8: Verified RouterReducer string-template already correct (TypeId value class toString)
2026-05-29 01:01:45 +04:00
kami eed557fe9c fix: address P0-4 chat zombie, P0-5 double SessionStarted, P0-6 steering wrong content 2026-05-28 23:10:29 +04:00
kami 57d2237ba0 fix: orchestrator race conditions, diff preview, session snapshot tools, and test fixes
- Fix duplicate inference requests after approval resume by registering orchestrator
  jobs in activeSessionJobs (ServerModule.launchSessionRun), so the guard in the
  OrchestrationResumedEvent subscription prevents spurious duplicate resume.
- Replace blocking Files.readString in computeToolPreview with withContext(Dispatchers.IO)
  by making the function suspend — no blocking I/O on the coroutine dispatcher.
- Guard orderedIds.add() in rebuildTools against duplicate invocation IDs on replayed
  ToolInvocationRequestedEvent to prevent duplicate tool records in snapshots.
- Add unified-diff preview for file_write tools: computeToolPreview reads existing
  file and generates ---/+++ diff before emitting ApprovalRequestedEvent.
- Render diff preview with color coding in ApprovalSurface (@@ yellow, +/- green/red).
- Include current-stage tool records in SessionSnapshot via rebuildTools event replay.
- Fix RouterContextBuilderTest: recursive buildPack helper and 2 tests using class-level
  builder instead of their local builder instances (conversationKeepLast mismatch).
- Add suspend keyword to RouterFacadeTest mock, fix buildPack recursion.
2026-05-28 15:37:16 +04:00
kami e05532e7b2 feat: implement CreateGrant handler and grant-aware approval engine
- GlobalStreamHandler.handleCreateGrant validates scope (SESSION/STAGE)
  and sends ProtocolError to client on validation failure instead of
  silent log.warn + drop.
- Derive event stageId directly from GrantScope type, removing
  redundant stageIdForEvent tuple.
- SessionOrchestrator integrates ApprovalEngine to check active grants
  before requesting user approval for T2+ tool calls.
- ContextEntry gains EntryRole (SYSTEM/USER/ASSISTANT/TOOL) field for
  proper chat message role mapping.
- RouterContextBuilder.build is now suspend; uses Tokenizer for
  accurate token estimation with fallback to character-based estimate.
- LlamaCppInferenceProvider maps EntryRole to ChatMessage role instead
  of heuristic layer-based inference.
2026-05-28 14:06:58 +04:00
kami ac05ad8733 chore(clean up): remove unused imports, unnecessary object impls, concurrency/coroutine issues, trailing commas, etc. 2026-05-24 22:56:19 +04:00
kami 70420083ac feat(server-02): promote EventStore to global-sequence store with subscribeAll
StoredEvent gains sessionSequence: Long (per-session monotonic); existing
sequence field becomes the global monotonic cursor across the entire store.
SqliteEventStore allocates both counters inside the same BEGIN IMMEDIATE
transaction and emits to a global MutableSharedFlow after commit.
InMemoryEventStore mirrors this with an AtomicLong global counter and a
suspend-on-overflow SharedFlow. EventStore interface gains subscribeAll()
and lastGlobalSequence(). Contract tests updated and extended to cover
global-sequence and cross-session subscribeAll invariants.
2026-05-24 22:23:43 +04:00
kami fc7b879891 feature(event-sourcing): baseline for the architecture review fixes.
- fixed the tool overlay in tui.
- fixed tool calling - added schemas.
- getting all sessionIds via event store.
- instead of sending all events on the replay - there's a new way for replaying.
- session snapshot is now a thing getting sent for previously created sessions.
- added logging to important parts.
- revert workflowId from WorkflowCompletedEvent.
2026-05-24 18:57:56 +04:00
kami f827685ed0 chore(damn): detekt, build, tests, formatting
fixed detekt issues where possible.
fixed disttar failing build because tools is added twice in the server module.
added workflowId where required.
fixed some tests not being recognized because of runBlocking without explicit return type.
formatting + imports.
2026-05-22 00:10:05 +04:00
kami 2c459da009 feat(router): implement Epic 14 — core:router module
Implements the full conversational router facade: RouterState, RouterReducer,
RouterProjector, RouterRepository, RouterContextBuilder, RouterFacade, protocol
types, WebSocket wiring, infrastructure factory, and deterministic test suite.

Also fixes spec divergences found in post-implementation review:
- Add SteeringNote domain object to core:context (epic prerequisite)
- Rename RouterFacade.handleChat → onUserInput per spec interface contract
- Add in-memory ConcurrentHashMap conversation history to DefaultRouterFacade
- Make RouterRepository.getRouterState suspend
- Rename RouterConfig.keepLast → conversationKeepLast, fix defaults (6, 4096)
- Refactor InfrastructureModule.createRouterFacade to self-assemble internally
- Fix FileReadTool: allowedPaths was dead constructor param (@SuppressUnusedParameter);
  now stored as private val and enforced in validateRequest
- Disable koverVerify on modules tested via testing/ submodules or with
  hardware/integration dependencies (24 modules); build gate now passes clean
2026-05-21 15:06:20 +04:00
kami 3365208d3a chore(cleanup): remove dead Module stubs and unused imports; restore deferred files
Deletes 21 Module.kt scaffolding objects that were never wired into any DI
registry. Removes unused imports across 8 production and 3 test files.

Restores StageExecutor, CyclePolicyResolver, PolicyValidation, and
ReplayContractTest — these are deferred features, not dead code.
2026-05-19 13:43:21 +04:00
kami d2518849d7 fix(apps): send workflowId not path from CLI; wire sandboxRoot and systemPromptPath in server; update prompts and fixtures 2026-05-18 21:40:17 +04:00
kami 219e2c762e feat(cas): content-addressed artifact store (steps 1–8)
New core:artifacts-store interface + infrastructure/artifacts-cas
implementation: segment files + SQLite index, Blake3 hashing,
group-commit fsync via flushBefore, recovery tail-scan, manual
compactor, and oldest-first disk-cap evictor.

Inference events now carry promptArtifactId / responseArtifactId;
orchestrators put bytes before emitting. SqliteEventStore wraps
its txn in artifactStore.flushBefore so segment data is fsynced
before the event commit, making the crash window non-corrupting
(TailScanner re-indexes orphan tail records on reopen).

Compactor and evictor are mutually exclusive via maintenanceMutex.
Step 9 (cloud sync) deferred to a later epic.

See docs/reviews/2026-05-18-cas-steps-1-8-review.md for the final
review.
2026-05-18 12:22:38 +04:00
kami 2207a37549 epic-13: add cli and tui entry point, finish epic. 2026-05-17 03:20:43 +04:00
kami ce723afc8b refactor(imports): optimize imports 2026-05-17 03:19:57 +04:00
kami c77277af0b epic-12: after epic audit and init commit 2026-05-17 03:19:39 +04:00