fix(events,tools,toolintent): failure attribution, one path normalization rule, file_copy (#713)

Three generic harness fixes from the web-ui postmortem dataset. Nothing here keys on a language, framework, build tool or task type. 1. Failure attribution. WorkflowFailedEvent carries one primary FailureAttribution (AGENT | HARNESS | WORKFLOW | ENVIRONMENT | PROVIDER | OPERATOR | UNKNOWN), defaulted to UNKNOWN so pre-field events replay unchanged. FailureAttributor is the deterministic reason->layer mapping, used both at emission and when classifying history, so the baseline and the live metric are one measurement. Emission sites set it: failWorkflow derives from the reason unless the caller knows the layer, cancellation is OPERATOR, the server catch-all falls back to HARNESS, a grounding-rejected plan is AGENT. Multi-cause chains stay on FailureTicketOpened — no second causal structure. GET /metrics/failure-attribution (FailureAttributionInspectionService, mirroring ToolReliabilityInspectionService) reports counts, share, UNKNOWN share, the preserved reasons and the ticket categories from the same sessions. Read-only: historical events are classified at READ time and reported as `inferred`, never written back over an append-only log. Baseline over the local log, 122 terminal failures: AGENT 51 (41.8%), OPERATOR 28 (23.0%), WORKFLOW 19 (15.6%), PROVIDER 15 (12.3%), HARNESS 6 (4.9%), ENVIRONMENT 3 (2.5%), UNKNOWN 0. 2. The `~` guard bug. ToolPath is now the ONE canonical normalization rule (expand a leading `~`/`~/`, keep absolutes, anchor relatives on the session working dir). Every filesystem tool, all six plane-2 path rules and the approval preview resolve through it, so policy and existence checks inspect the path the tool will operate on. `~/.gradle/init.d/offline.gradle` used to resolve to `<workspace>/~/.gradle/...`: reported non-existent AND in-workspace, so the reference gate called a real file a hallucination and the out-of-workspace prompt never fired. Containment and external-read approval behaviour are unchanged — the expanded path is simply outside the workspace, where it always belonged. 3. file_copy (#713). A first-class tool with the writer's jail, tier, receipt, replay and CAS pre/post images; static and binary assets no longer move through the model's token stream. Needed one generic split: ParamRole.SOURCE_PATH marks a path a call reads FROM, so containment gates judge both params while write-target gates (read-before-write, stale-write, write scope, write manifest) judge the mutated one. ReadBeforeWriteRule exempts any call declaring a SOURCE_PATH: its content comes from disk, not from memory, and requiring a read of a binary is unsatisfiable. Existing tools declare no SOURCE_PATH, so their behaviour is byte-identical. Tests: ToolPathTest (9), FailureAttributionTest (10), PathNormalizationRuleTest (6), FileCopyToolTest (10), plus a home-relative FileReadTool read. ./gradlew check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 11:57:49 +04:00
parent 519290368f
commit 6a8a7b31c1
36 changed files with 1232 additions and 79 deletions
@@ -19,6 +19,7 @@ All sources under `apps/server/src/`.
 - `GET /health` — health report (probes: event-store, llama-server, disk watermark)
 - `GET /stats` — metrics report (MetricsProjection)
 - `GET /metrics/tool-reliability` — per-model tool-call validity across the event log (`ToolReliabilityInspectionService`); groundwork for capability-aware routing
+- `GET /metrics/failure-attribution` — terminal-failure attribution across the event log (`FailureAttributionInspectionService`): count and share per `FailureAttribution` layer, the UNKNOWN share, the preserved reasons behind each row, and the `FailureTicketOpened` categories from the same sessions. Read-only: events recorded before `WorkflowFailedEvent.attribution` existed are classified at read time by `FailureAttributor` and reported as `inferred`, never written back.
 - Optional `[git]` transport creates `run/<sessionId>` from a server-local checkout and pushes it at terminal state; clients review with ordinary Git and never supply a remote URL as `cwd`.
 - Repo-map L3 embeddings use bounded, recorded source descriptors (module/package, imports, leading purpose comment, symbols); raw file bodies are never embedded. Their versioned `repomap:v2` namespace forces a one-time re-embed when the semantic document format changes.
 - At boot, `tools.workspace_root` is the authoritative default tool jail. Every session records its own resolved workspace binding; repo maps, project memory, profile/instruction snapshots, and git run branches use that binding and skip unbound sessions. `[project]` never supplies a workspace root.
@@ -1,6 +1,7 @@
 package com.correx.apps.server

 import com.correx.apps.server.health.HealthInspectionService
+import com.correx.apps.server.metrics.FailureAttributionInspectionService
 import com.correx.apps.server.metrics.ToolReliabilityInspectionService
 import com.correx.apps.server.routes.providerRoutes
 import com.correx.apps.server.routes.sessionRoutes
@@ -40,6 +41,7 @@ fun Application.configureServer(module: ServerModule) {
    val globalStreamHandler = GlobalStreamHandler(module)
    val healthInspection = HealthInspectionService(module.eventStore)
    val toolReliability = ToolReliabilityInspectionService(module.eventStore)
+    val failureAttribution = FailureAttributionInspectionService(module.eventStore)

    routing {
        get("/health") {
@@ -59,6 +61,13 @@ fun Application.configureServer(module: ServerModule) {
            call.respond(toolReliability.inspect())
        }

+        // Terminal-failure attribution across the whole event log: which layer each WorkflowFailed
+        // belongs to, and the UNKNOWN share. Read-only; events recorded before the attribution field
+        // are classified at read time and reported as `inferred`, never written back.
+        get("/metrics/failure-attribution") {
+            call.respond(failureAttribution.inspect())
+        }
+
        webSocket("/stream") {
            globalStreamHandler.handle(this)
        }
@@ -8,6 +8,8 @@ import com.correx.apps.server.registry.ProviderRegistry
 import com.correx.apps.server.registry.WorkflowRegistry
 import com.correx.apps.server.workspace.WorkspaceResolver
 import com.correx.apps.server.workspace.WorkspaceResolution
+import com.correx.core.events.events.FailureAttribution
+import com.correx.core.events.events.FailureAttributor
 import com.correx.core.events.events.SessionWorkspaceBoundEvent
 import com.correx.core.kernel.orchestration.WorkspaceContext
 import com.correx.core.approvals.ApprovalProjector
@@ -478,6 +480,10 @@ class ServerModule(
                    stageId = failingStageId,
                    reason = reason,
                    retryExhausted = retryExhausted,
+                    // This is the catch-all for a throwable that escaped the orchestrator, so the
+                    // default layer is correx itself; the reason text still wins when it names an
+                    // outer layer (provider timeout, missing program, operator cancellation).
+                    attribution = FailureAttributor.classify(reason, fallback = FailureAttribution.HARNESS),
                ),
            ),
        )
@@ -7,6 +7,7 @@ import com.correx.core.events.events.CapabilityGapVerdict
 import com.correx.core.events.events.EventMetadata
 import com.correx.core.events.events.ExecutionPlanLockedEvent
 import com.correx.core.events.events.ExecutionPlanRejectedEvent
+import com.correx.core.events.events.FailureAttribution
 import com.correx.core.events.events.NewEvent
 import com.correx.core.events.events.PlanGroundingEvaluatedEvent
 import com.correx.core.events.events.PlanGroundingVerdict
@@ -371,6 +372,8 @@ class FreestyleDriver(
                    stageId = StageId("architect"),
                    reason = "execution plan rejected ($source): $reason",
                    retryExhausted = false,
+                    // The rejected plan is the model's own output; the harness evaluated it correctly.
+                    attribution = FailureAttribution.AGENT,
                ),
            ),
        )
@@ -0,0 +1,140 @@
+package com.correx.apps.server.metrics
+
+import com.correx.core.events.events.FailureAttribution
+import com.correx.core.events.events.FailureAttributor
+import com.correx.core.events.events.FailureTicketOpenedEvent
+import com.correx.core.events.events.WorkflowFailedEvent
+import com.correx.core.events.stores.EventStore
+import kotlinx.serialization.Serializable
+
+private const val PERCENT = 100.0
+private const val REASON_BUCKET_MAX = 110
+private const val TOP_REASONS = 10
+
+@Serializable
+data class AttributionRow(
+    val attribution: String,
+    val count: Long,
+    val sharePct: Double,
+    /** Failures whose event carried this attribution when it was recorded. */
+    val recorded: Long,
+    /** Failures classified from the preserved reason at read time (the field was absent/UNKNOWN). */
+    val inferred: Long,
+    /** The preserved [WorkflowFailedEvent.reason] texts behind this row, most frequent first. */
+    val topReasons: List<ReasonCount>,
+    /** Categories of the [FailureTicketOpenedEvent]s in the same sessions — the causal chain when
+     *  more than one layer contributed. Empty when no tickets were opened. */
+    val contributingTicketCategories: List<ReasonCount>,
+)
+
+@Serializable
+data class FailureAttributionReport(
+    val totalFailures: Long,
+    val recordedFailures: Long,
+    val inferredFailures: Long,
+    val unknownCount: Long,
+    val unknownPct: Double,
+    val byAttribution: List<AttributionRow>,
+    /** Every reason that no marker matched, so UNKNOWN can never sit unexamined. */
+    val unknownReasons: List<ReasonCount>,
+)
+
+/**
+ * Failure attribution across the whole event log: how many terminal [WorkflowFailedEvent]s belong to
+ * each layer, and what share is still UNKNOWN. The denominator you need before changing execution
+ * behaviour — "correx failed N runs" is not actionable, "N of them were harness defects" is.
+ *
+ * Read-only and idempotent by construction: historical events recorded before
+ * [WorkflowFailedEvent.attribution] existed are classified at READ time by [FailureAttributor], the
+ * same function live emission uses. Nothing is written back — history is append-only, and a
+ * re-derived classification is a projection, not a fact (Hard Invariant #2). Each row separates what
+ * was `recorded` at emission from what this service `inferred`, so a backfilled baseline never
+ * masquerades as originally-recorded data. Re-running it over the same log yields the same numbers.
+ */
+class FailureAttributionInspectionService(private val eventStore: EventStore) {
+
+    private class Agg {
+        var recorded: Long = 0
+        var inferred: Long = 0
+        val reasons: MutableMap<String, Long> = linkedMapOf()
+        val sessions: MutableSet<String> = linkedSetOf()
+    }
+
+    @Suppress("NestedBlockDepth")
+    fun inspect(): FailureAttributionReport {
+        val byAttribution = linkedMapOf<FailureAttribution, Agg>()
+        val ticketsBySession = linkedMapOf<String, MutableMap<String, Long>>()
+        val unknownReasons = linkedMapOf<String, Long>()
+
+        eventStore.allEvents().forEach { stored ->
+            when (val payload = stored.payload) {
+                is FailureTicketOpenedEvent -> {
+                    val categories = ticketsBySession.getOrPut(payload.sessionId.value) { linkedMapOf() }
+                    categories[payload.category] = (categories[payload.category] ?: 0) + 1
+                }
+
+                is WorkflowFailedEvent -> {
+                    val wasRecorded = payload.attribution != FailureAttribution.UNKNOWN
+                    val attribution =
+                        if (wasRecorded) payload.attribution else FailureAttributor.classify(payload.reason)
+                    val agg = byAttribution.getOrPut(attribution) { Agg() }
+                    if (wasRecorded) agg.recorded++ else agg.inferred++
+                    val bucket = payload.reason.lineSequence().firstOrNull().orEmpty().take(REASON_BUCKET_MAX)
+                    agg.reasons[bucket] = (agg.reasons[bucket] ?: 0) + 1
+                    agg.sessions += payload.sessionId.value
+                    if (attribution == FailureAttribution.UNKNOWN) {
+                        unknownReasons[bucket] = (unknownReasons[bucket] ?: 0) + 1
+                    }
+                }
+
+                else -> Unit
+            }
+        }
+
+        val total = byAttribution.values.sumOf { it.recorded + it.inferred }
+        val unknown = byAttribution[FailureAttribution.UNKNOWN]?.let { it.recorded + it.inferred } ?: 0
+        val rows = byAttribution.entries
+            .sortedByDescending { it.value.recorded + it.value.inferred }
+            .map { (attribution, agg) -> row(attribution, agg, total, ticketsBySession) }
+
+        return FailureAttributionReport(
+            totalFailures = total,
+            recordedFailures = byAttribution.values.sumOf { it.recorded },
+            inferredFailures = byAttribution.values.sumOf { it.inferred },
+            unknownCount = unknown,
+            unknownPct = share(unknown, total),
+            byAttribution = rows,
+            unknownReasons = topOf(unknownReasons),
+        )
+    }
+
+    private fun row(
+        attribution: FailureAttribution,
+        agg: Agg,
+        total: Long,
+        ticketsBySession: Map<String, Map<String, Long>>,
+    ): AttributionRow {
+        val count = agg.recorded + agg.inferred
+        val tickets = linkedMapOf<String, Long>()
+        agg.sessions.forEach { sessionId ->
+            ticketsBySession[sessionId]?.forEach { (category, n) ->
+                tickets[category] = (tickets[category] ?: 0) + n
+            }
+        }
+        return AttributionRow(
+            attribution = attribution.name,
+            count = count,
+            sharePct = share(count, total),
+            recorded = agg.recorded,
+            inferred = agg.inferred,
+            topReasons = topOf(agg.reasons),
+            contributingTicketCategories = topOf(tickets),
+        )
+    }
+
+    private fun topOf(counts: Map<String, Long>): List<ReasonCount> =
+        counts.entries.sortedByDescending { it.value }.take(TOP_REASONS).map { ReasonCount(it.key, it.value) }
+
+    private fun share(part: Long, total: Long): Double =
+        if (total == 0L) 0.0 else part.toDouble() / total * PERCENT
+}