fix(events,tools,toolintent): failure attribution, one path normalization rule, file_copy (#713)
Three generic harness fixes from the web-ui postmortem dataset. Nothing here keys on a language, framework, build tool or task type. 1. Failure attribution. WorkflowFailedEvent carries one primary FailureAttribution (AGENT | HARNESS | WORKFLOW | ENVIRONMENT | PROVIDER | OPERATOR | UNKNOWN), defaulted to UNKNOWN so pre-field events replay unchanged. FailureAttributor is the deterministic reason->layer mapping, used both at emission and when classifying history, so the baseline and the live metric are one measurement. Emission sites set it: failWorkflow derives from the reason unless the caller knows the layer, cancellation is OPERATOR, the server catch-all falls back to HARNESS, a grounding-rejected plan is AGENT. Multi-cause chains stay on FailureTicketOpened — no second causal structure. GET /metrics/failure-attribution (FailureAttributionInspectionService, mirroring ToolReliabilityInspectionService) reports counts, share, UNKNOWN share, the preserved reasons and the ticket categories from the same sessions. Read-only: historical events are classified at READ time and reported as `inferred`, never written back over an append-only log. Baseline over the local log, 122 terminal failures: AGENT 51 (41.8%), OPERATOR 28 (23.0%), WORKFLOW 19 (15.6%), PROVIDER 15 (12.3%), HARNESS 6 (4.9%), ENVIRONMENT 3 (2.5%), UNKNOWN 0. 2. The `~` guard bug. ToolPath is now the ONE canonical normalization rule (expand a leading `~`/`~/`, keep absolutes, anchor relatives on the session working dir). Every filesystem tool, all six plane-2 path rules and the approval preview resolve through it, so policy and existence checks inspect the path the tool will operate on. `~/.gradle/init.d/offline.gradle` used to resolve to `<workspace>/~/.gradle/...`: reported non-existent AND in-workspace, so the reference gate called a real file a hallucination and the out-of-workspace prompt never fired. Containment and external-read approval behaviour are unchanged — the expanded path is simply outside the workspace, where it always belonged. 3. file_copy (#713). A first-class tool with the writer's jail, tier, receipt, replay and CAS pre/post images; static and binary assets no longer move through the model's token stream. Needed one generic split: ParamRole.SOURCE_PATH marks a path a call reads FROM, so containment gates judge both params while write-target gates (read-before-write, stale-write, write scope, write manifest) judge the mutated one. ReadBeforeWriteRule exempts any call declaring a SOURCE_PATH: its content comes from disk, not from memory, and requiring a read of a binary is unsatisfiable. Existing tools declare no SOURCE_PATH, so their behaviour is byte-identical. Tests: ToolPathTest (9), FailureAttributionTest (10), PathNormalizationRuleTest (6), FileCopyToolTest (10), plus a home-relative FileReadTool read. ./gradlew check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -20,6 +20,7 @@ CORREX kernel team. This is the most cross-cutting module in the codebase — ch
|
||||
- `JsonEventSerializer` / `EventSerializer` — serialize/deserialize `StoredEvent` to JSON.
|
||||
- `EventDispatcher` — broadcasts events to in-process listeners.
|
||||
- Domain event files: `ApprovalEvents`, `ArtifactEvents`, `ContextEvents`, `InferenceEvents`, `OrchestrationEvents`, `RouterEvents`, `SessionEvents`, `TaskEvents`, `ToolEvents`, `IntentEvents`, `RiskAssessedEvent`, `JournalCompactedEvent`, and many more — all payload definitions live here.
|
||||
- `FailureAttribution` / `FailureAttributor` — the terminal-failure taxonomy (`AGENT`, `HARNESS`, `WORKFLOW`, `ENVIRONMENT`, `PROVIDER`, `OPERATOR`, `UNKNOWN`) carried as `WorkflowFailedEvent.attribution`, plus the deterministic reason→layer mapping used both at emission and when classifying historical events. One primary attribution per terminal event; a multi-cause chain is the session's `FailureTicketOpenedEvent`s, not a second structure.
|
||||
- `LspDiagnosticsCompletedEvent` records pulled language-server diagnostics or a graceful skip reason; replay consumes this observation and never contacts the server.
|
||||
- Shared vocabulary: `IdentityTypes` (SessionId, TaskId, etc.), `Tier`, `TokenUsage`, `ToolReceipt`, `ToolRequest`, `RiskLevel`, `RetryPolicy`, `GrantScope`, `GrantLedger`.
|
||||
|
||||
@@ -30,7 +31,8 @@ CORREX kernel team. This is the most cross-cutting module in the codebase — ch
|
||||
- Event classes are `@Serializable data class` with no mutable state. No methods beyond data accessors.
|
||||
- `RunBranchPushedEvent` records an optional server Git transport push only after it succeeds; its branch/base/head SHAs are observations, not values replay recalculates.
|
||||
- `RepoMapEntry.descriptor` is a bounded source-purpose observation recorded with the repo map and used when constructing semantic L3 embeddings.
|
||||
- Do not add domain logic to events. They are records, not actors.
|
||||
- Do not add domain logic to events. They are records, not actors. `FailureAttributor` is the one exception by design: a pure reason→layer function that must be identical for live emission and for historical classification, so it lives beside the enum it returns.
|
||||
- `WorkflowFailedEvent.attribution` defaults to `UNKNOWN` so pre-field events replay unchanged. Classify those at READ time (see `FailureAttributionInspectionService`); never rewrite history to backfill them.
|
||||
- `EgressAllowlistProjection` — special projection kept in this module because it is used by both `core:toolintent` and `core:events` consumers; it is a shared cross-cutting projection.
|
||||
|
||||
## Verification
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
package com.correx.core.events.events
|
||||
|
||||
import kotlinx.serialization.Serializable
|
||||
|
||||
/**
|
||||
* WHOSE failure a terminal [WorkflowFailedEvent] was: the primary layer that has to change for the
|
||||
* run to succeed. One value per terminal event. When several causes contributed, the causal chain is
|
||||
* the session's [FailureTicketOpenedEvent]s — this enum does not model chains.
|
||||
*
|
||||
* The point is measurement: "correx failed 99 runs" is not actionable, "61 of them were harness
|
||||
* defects" is. Read the layer, not the symptom.
|
||||
*/
|
||||
@Serializable
|
||||
enum class FailureAttribution {
|
||||
/** The model produced invalid work while the harness operated correctly. */
|
||||
AGENT,
|
||||
|
||||
/** Correx's own runtime: a linkage error, a bug in a reducer/tool layer, a false observation
|
||||
* handed to the agent, or an expectation correx could not evaluate for lack of instrumentation. */
|
||||
HARNESS,
|
||||
|
||||
/** The workflow/graph definition: no transition matched, a condition referenced a field that
|
||||
* cannot exist, a declared prompt or stage was never authored. */
|
||||
WORKFLOW,
|
||||
|
||||
/** The machine the run executes on: a missing executable, permissions, disk, ports. */
|
||||
ENVIRONMENT,
|
||||
|
||||
/** The inference provider: unavailable, timed out, or answered with a body correx cannot read. */
|
||||
PROVIDER,
|
||||
|
||||
/** A human ended the run: cancellation, or a denied/rejected approval. */
|
||||
OPERATOR,
|
||||
|
||||
/** Not classifiable from the recorded reason. A metric, not a bucket: a rising UNKNOWN share
|
||||
* means the taxonomy or the reason text needs work, and every UNKNOWN is a defect to triage. */
|
||||
UNKNOWN,
|
||||
}
|
||||
|
||||
/**
|
||||
* Deterministic mapping from a terminal failure reason to its [FailureAttribution].
|
||||
*
|
||||
* Same function for live emission and for classifying historical events recorded before the field
|
||||
* existed, so a backfilled baseline and a live metric are the same measurement. It is a pure
|
||||
* function of the reason string: no clock, no I/O, no session lookup — safe to re-run over the whole
|
||||
* event log any number of times.
|
||||
*
|
||||
* Markers are matched in layer order — OPERATOR, PROVIDER, ENVIRONMENT, WORKFLOW, HARNESS, AGENT —
|
||||
* because the outermost cause wins: a provider timeout that surfaces as an artifact-validation
|
||||
* failure is still a provider failure. Match on what the layer says about ITSELF (a provider being
|
||||
* unavailable, a program that cannot be run), never on the domain of the run: nothing here may key
|
||||
* on a language, framework, build tool or task type.
|
||||
*/
|
||||
object FailureAttributor {
|
||||
|
||||
/**
|
||||
* Classifies [reason]. [fallback] is returned when no marker matches — a call site that knows
|
||||
* the layer from its position (e.g. a top-level catch-all in the correx runtime) supplies its
|
||||
* own instead of leaving the failure [FailureAttribution.UNKNOWN].
|
||||
*/
|
||||
fun classify(reason: String, fallback: FailureAttribution = FailureAttribution.UNKNOWN): FailureAttribution {
|
||||
val text = reason.lowercase()
|
||||
// A reason that is a bare JVM binary name with no prose is a linkage/classload error
|
||||
// (NoClassDefFoundError.getMessage()), i.e. a correx runtime defect.
|
||||
if (text.isNotBlank() && !text.contains(' ') && text.contains('/') && !text.contains('.')) {
|
||||
return FailureAttribution.HARNESS
|
||||
}
|
||||
return MARKERS.firstOrNull { (_, markers) -> markers.any { it in text } }?.first ?: fallback
|
||||
}
|
||||
|
||||
private val MARKERS: List<Pair<FailureAttribution, List<String>>> = listOf(
|
||||
FailureAttribution.OPERATOR to listOf(
|
||||
"cancelled",
|
||||
"canceled",
|
||||
"approval denied",
|
||||
"approval rejected",
|
||||
"rejected by operator",
|
||||
),
|
||||
FailureAttribution.PROVIDER to listOf(
|
||||
"is unavailable",
|
||||
"health check failed",
|
||||
"connection refused",
|
||||
"request timeout has expired",
|
||||
"no provider satisfies",
|
||||
"returned 400",
|
||||
"returned 401",
|
||||
"returned 403",
|
||||
"returned 404",
|
||||
"returned 5",
|
||||
"chatcompletionresponse",
|
||||
"no completion returned",
|
||||
"context window exceeded",
|
||||
),
|
||||
FailureAttribution.ENVIRONMENT to listOf(
|
||||
"cannot run program",
|
||||
"exec failed",
|
||||
"command not found",
|
||||
"permission denied",
|
||||
"no space left",
|
||||
"address already in use",
|
||||
),
|
||||
FailureAttribution.WORKFLOW to listOf(
|
||||
"no transition condition matched",
|
||||
"no matching transition",
|
||||
"condition evaluation failed",
|
||||
// A stage's declaration disagrees with reality: the prerequisite it names is unresolved
|
||||
// or sits outside the scope it declared. Both are authoring defects in the definition.
|
||||
"build prerequisite",
|
||||
"declared prompt",
|
||||
"unknown stage",
|
||||
"no such stage",
|
||||
),
|
||||
FailureAttribution.HARNESS to listOf(
|
||||
"noclassdeffounderror",
|
||||
"nosuchmethod",
|
||||
"classnotfound",
|
||||
"not supported in map",
|
||||
"hex string must have even length",
|
||||
"could not be evaluated",
|
||||
"no instrumentation",
|
||||
"unexpected orchestrator failure",
|
||||
),
|
||||
FailureAttribution.AGENT to listOf(
|
||||
"did not produce declared artifacts",
|
||||
"did not satisfy its file contract",
|
||||
"did not pass",
|
||||
"failed semantic review",
|
||||
"declared no artifacts",
|
||||
"review loop exhausted",
|
||||
// A call plane-2 denied: the harness evaluated policy correctly, the agent proposed it.
|
||||
"blocked by tool-call policy",
|
||||
"validation failed",
|
||||
"artifact repair failed",
|
||||
"repair ladder exhausted",
|
||||
"recovery route budget exhausted",
|
||||
"refinement loop",
|
||||
"execution plan rejected",
|
||||
"is stuck",
|
||||
"failed to decode",
|
||||
),
|
||||
)
|
||||
}
|
||||
@@ -34,6 +34,12 @@ data class WorkflowFailedEvent(
|
||||
val stageId: StageId,
|
||||
val reason: String,
|
||||
val retryExhausted: Boolean,
|
||||
// WHOSE failure this was — the layer that must change for the run to succeed (see
|
||||
// [FailureAttribution]). Set at emission by the site that knows the cause, or derived from
|
||||
// [reason] by [FailureAttributor]; the original [reason] is always preserved alongside it.
|
||||
// Defaulted to UNKNOWN so events recorded before this field replay unchanged: a classification
|
||||
// for those is INFERRED at read time, never written back over history.
|
||||
val attribution: FailureAttribution = FailureAttribution.UNKNOWN,
|
||||
) : EventPayload
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
package com.correx.core.events.events
|
||||
|
||||
import com.correx.core.events.types.SessionId
|
||||
import com.correx.core.events.types.StageId
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
|
||||
/**
|
||||
* The taxonomy's contract. Cases are drawn from real `WorkflowFailed.reason` texts in the local
|
||||
* event log so the mapping is checked against failures that actually happened, and every case keys
|
||||
* on what a LAYER says about itself — never on a language, framework or build tool.
|
||||
*/
|
||||
class FailureAttributionTest {
|
||||
|
||||
private fun assertLayer(expected: FailureAttribution, reason: String) =
|
||||
assertEquals(expected, FailureAttributor.classify(reason), reason)
|
||||
|
||||
@Test
|
||||
fun `operator-ended runs`() {
|
||||
assertLayer(FailureAttribution.OPERATOR, "CANCELLED")
|
||||
assertLayer(FailureAttribution.OPERATOR, "approval denied")
|
||||
assertLayer(FailureAttribution.OPERATOR, "approval rejected for stage architect")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `provider failures`() {
|
||||
assertLayer(
|
||||
FailureAttribution.PROVIDER,
|
||||
"Provider 'llama-cpp:default' is unavailable: Health check failed: Connection refused",
|
||||
)
|
||||
assertLayer(
|
||||
FailureAttribution.PROVIDER,
|
||||
"Request timeout has expired [url=http://127.0.0.1:10000/v1/chat/completions, " +
|
||||
"request_timeout=600000 ms]",
|
||||
)
|
||||
assertLayer(FailureAttribution.PROVIDER, "No provider satisfies capabilities [] for stage 'routing'")
|
||||
// A provider body correx cannot decode is a provider-communication failure, not a bad artifact.
|
||||
assertLayer(
|
||||
FailureAttribution.PROVIDER,
|
||||
"Illegal input: Fields [id, choices, usage] are required for type with serial name " +
|
||||
"'com.correx.infrastructure.inference.llama.cpp.ChatCompletionResponse', but they were missing",
|
||||
)
|
||||
assertLayer(FailureAttribution.PROVIDER, "llama-server returned 400 Bad Request: {\"error\":{}}")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `environment failures`() {
|
||||
assertLayer(
|
||||
FailureAttribution.ENVIRONMENT,
|
||||
"Cannot run program \"cd\" (in directory \"/w\"): Exec failed, error: 2 (No such file or directory)",
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `workflow-definition failures`() {
|
||||
assertLayer(FailureAttribution.WORKFLOW, "no transition condition matched from stage analyst")
|
||||
assertLayer(
|
||||
FailureAttribution.WORKFLOW,
|
||||
"condition evaluation failed on 'verify_completion->done': Field 'verdict' not found",
|
||||
)
|
||||
assertLayer(FailureAttribution.WORKFLOW, "[SessionOrchestrator] stage=analyst: declared prompt 'x' missing")
|
||||
assertLayer(FailureAttribution.WORKFLOW, "no matching transition from stage A")
|
||||
assertLayer(FailureAttribution.WORKFLOW, "build prerequisite 'x' unresolved after bootstrap: missing")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `harness failures`() {
|
||||
// A bare JVM binary name with no prose is a linkage error inside correx itself.
|
||||
assertLayer(
|
||||
FailureAttribution.HARNESS,
|
||||
"com/correx/core/kernel/orchestration/SessionOrchestrator\$failWorkflow\$1",
|
||||
)
|
||||
assertLayer(FailureAttribution.HARNESS, "com/correx/core/approvals/GrantLedgerKt")
|
||||
assertLayer(FailureAttribution.HARNESS, "null values are not supported in Map<String, Any>")
|
||||
// An expectation correx could not evaluate for lack of instrumentation is ours, not the agent's.
|
||||
assertLayer(FailureAttribution.HARNESS, "expected_result could not be evaluated: no instrumentation")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `agent failures`() {
|
||||
assertLayer(FailureAttribution.AGENT, "stage implementer did not produce declared artifacts: patch")
|
||||
assertLayer(FailureAttribution.AGENT, "validation failed")
|
||||
assertLayer(FailureAttribution.AGENT, "artifact repair failed (FORMATTING): could not extract a JSON object")
|
||||
assertLayer(FailureAttribution.AGENT, "refinement loop 'implementer->reviewer' exceeded 2 iterations")
|
||||
assertLayer(FailureAttribution.AGENT, "recovery route budget exhausted for stage ui_review (gate=execution)")
|
||||
assertLayer(FailureAttribution.AGENT, "repair ladder exhausted for stage x (gate=stage_loop_break)")
|
||||
assertLayer(FailureAttribution.AGENT, "execution plan rejected (grounding): plan failed grounding")
|
||||
assertLayer(FailureAttribution.AGENT, "stage x did not satisfy its file contract. Fix these before review:")
|
||||
assertLayer(FailureAttribution.AGENT, "stage x did not pass its PROJECT build gate")
|
||||
assertLayer(FailureAttribution.AGENT, "stage x did not pass static analysis. Fix these before review:")
|
||||
assertLayer(FailureAttribution.AGENT, "stage x failed semantic review — fix these correctness issues:")
|
||||
assertLayer(FailureAttribution.AGENT, "stage x declared no artifacts and ran no tools")
|
||||
assertLayer(FailureAttribution.AGENT, "review loop exhausted after exactly 3 cycles.")
|
||||
assertLayer(FailureAttribution.AGENT, "blocked by tool-call policy")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `an unmatched reason is UNKNOWN, and a call site may supply its own fallback`() {
|
||||
assertLayer(FailureAttribution.UNKNOWN, "something nobody has seen before")
|
||||
assertEquals(
|
||||
FailureAttribution.HARNESS,
|
||||
FailureAttributor.classify("something nobody has seen before", FailureAttribution.HARNESS),
|
||||
)
|
||||
// The reason text still wins over a call site's fallback when it names an outer layer.
|
||||
assertEquals(
|
||||
FailureAttribution.OPERATOR,
|
||||
FailureAttributor.classify("CANCELLED", FailureAttribution.HARNESS),
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `classification is a pure function of the reason`() {
|
||||
val reason = "no transition condition matched from stage analyst"
|
||||
assertEquals(FailureAttributor.classify(reason), FailureAttributor.classify(reason))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `an event recorded before the field replays as UNKNOWN with its reason preserved`() {
|
||||
val stored = """{"sessionId":"s","stageId":"st","reason":"CANCELLED","retryExhausted":false}"""
|
||||
val event = Json.decodeFromString<WorkflowFailedEvent>(stored)
|
||||
assertEquals(FailureAttribution.UNKNOWN, event.attribution)
|
||||
assertEquals("CANCELLED", event.reason)
|
||||
// …and the historical baseline classifies it at read time, without rewriting history.
|
||||
assertEquals(FailureAttribution.OPERATOR, FailureAttributor.classify(event.reason))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a live event carries its attribution through a round-trip`() {
|
||||
val event = WorkflowFailedEvent(
|
||||
sessionId = SessionId("s"),
|
||||
stageId = StageId("st"),
|
||||
reason = "no transition condition matched from stage analyst",
|
||||
retryExhausted = true,
|
||||
attribution = FailureAttribution.WORKFLOW,
|
||||
)
|
||||
val json = Json.encodeToString(WorkflowFailedEvent.serializer(), event)
|
||||
assertEquals(event, Json.decodeFromString(WorkflowFailedEvent.serializer(), json))
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user