fix(events,tools,toolintent): failure attribution, one path normalization rule, file_copy (#713)
Three generic harness fixes from the web-ui postmortem dataset. Nothing here keys on a language, framework, build tool or task type. 1. Failure attribution. WorkflowFailedEvent carries one primary FailureAttribution (AGENT | HARNESS | WORKFLOW | ENVIRONMENT | PROVIDER | OPERATOR | UNKNOWN), defaulted to UNKNOWN so pre-field events replay unchanged. FailureAttributor is the deterministic reason->layer mapping, used both at emission and when classifying history, so the baseline and the live metric are one measurement. Emission sites set it: failWorkflow derives from the reason unless the caller knows the layer, cancellation is OPERATOR, the server catch-all falls back to HARNESS, a grounding-rejected plan is AGENT. Multi-cause chains stay on FailureTicketOpened — no second causal structure. GET /metrics/failure-attribution (FailureAttributionInspectionService, mirroring ToolReliabilityInspectionService) reports counts, share, UNKNOWN share, the preserved reasons and the ticket categories from the same sessions. Read-only: historical events are classified at READ time and reported as `inferred`, never written back over an append-only log. Baseline over the local log, 122 terminal failures: AGENT 51 (41.8%), OPERATOR 28 (23.0%), WORKFLOW 19 (15.6%), PROVIDER 15 (12.3%), HARNESS 6 (4.9%), ENVIRONMENT 3 (2.5%), UNKNOWN 0. 2. The `~` guard bug. ToolPath is now the ONE canonical normalization rule (expand a leading `~`/`~/`, keep absolutes, anchor relatives on the session working dir). Every filesystem tool, all six plane-2 path rules and the approval preview resolve through it, so policy and existence checks inspect the path the tool will operate on. `~/.gradle/init.d/offline.gradle` used to resolve to `<workspace>/~/.gradle/...`: reported non-existent AND in-workspace, so the reference gate called a real file a hallucination and the out-of-workspace prompt never fired. Containment and external-read approval behaviour are unchanged — the expanded path is simply outside the workspace, where it always belonged. 3. file_copy (#713). A first-class tool with the writer's jail, tier, receipt, replay and CAS pre/post images; static and binary assets no longer move through the model's token stream. Needed one generic split: ParamRole.SOURCE_PATH marks a path a call reads FROM, so containment gates judge both params while write-target gates (read-before-write, stale-write, write scope, write manifest) judge the mutated one. ReadBeforeWriteRule exempts any call declaring a SOURCE_PATH: its content comes from disk, not from memory, and requiring a read of a binary is unsatisfiable. Existing tools declare no SOURCE_PATH, so their behaviour is byte-identical. Tests: ToolPathTest (9), FailureAttributionTest (10), PathNormalizationRuleTest (6), FileCopyToolTest (10), plus a home-relative FileReadTool read. ./gradlew check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+3
-4
@@ -7,6 +7,7 @@ import com.correx.core.toolintent.ToolCallAssessmentInput
|
||||
import com.correx.core.toolintent.ToolCallRule
|
||||
import com.correx.core.toolintent.maxAction
|
||||
import com.correx.core.tools.contract.ToolCapability
|
||||
import com.correx.core.tools.contract.ToolPath
|
||||
import com.correx.core.validation.model.ValidationIssue
|
||||
import com.correx.core.validation.model.ValidationSeverity
|
||||
import java.nio.file.FileSystems
|
||||
@@ -52,10 +53,8 @@ class ManifestContainmentRule : ToolCallRule {
|
||||
val observations = mutableListOf<ToolCallObservation>()
|
||||
var disposition = RiskAction.PROCEED
|
||||
|
||||
for (raw in candidatePathStrings(input.paramRoles, input.request.parameters)) {
|
||||
val candidate = Path.of(raw)
|
||||
val resolvedInput =
|
||||
if (candidate.isAbsolute) candidate else input.workspace.workspaceRoot.resolve(candidate)
|
||||
for (raw in writeTargetPathStrings(input.paramRoles, input.request.parameters)) {
|
||||
val resolvedInput = ToolPath.resolve(raw, input.workspace.workspaceRoot)
|
||||
val resolvedReal = input.probe.resolveReal(resolvedInput)
|
||||
val inWorkspace = resolvedReal.startsWith(workspaceReal)
|
||||
val relative = if (inWorkspace) workspaceReal.relativize(resolvedReal).toString() else raw
|
||||
|
||||
+24
-6
@@ -14,13 +14,31 @@ internal fun extractParamStrings(value: Any?): List<String> = when (value) {
|
||||
}
|
||||
|
||||
/**
|
||||
* The path-like argument strings of a tool call: the values of params declared
|
||||
* [ParamRole.PATH], or — when none are declared — any string value that looks like a
|
||||
* path. Shared by the path-containment and write-manifest rules so both judge exactly
|
||||
* the same set of targets.
|
||||
* Every path-like argument of a tool call — the call's own targets ([ParamRole.PATH]) and any path
|
||||
* it merely reads from ([ParamRole.SOURCE_PATH]). Used by the containment/existence gates, which
|
||||
* must judge both: a source outside the workspace is as much an escape as a destination outside it.
|
||||
*/
|
||||
internal fun candidatePathStrings(paramRoles: Map<String, ParamRole>, parameters: Map<String, Any>): List<String> {
|
||||
val declared = paramRoles.filterValues { it == ParamRole.PATH }.keys
|
||||
internal fun candidatePathStrings(paramRoles: Map<String, ParamRole>, parameters: Map<String, Any>): List<String> =
|
||||
pathStringsForRoles(paramRoles, parameters, setOf(ParamRole.PATH, ParamRole.SOURCE_PATH))
|
||||
|
||||
/**
|
||||
* Only the paths a tool call MUTATES ([ParamRole.PATH]). Used by the write-target gates
|
||||
* (read-before-write, stale-write, write scope, write manifest): blocking a copy because its
|
||||
* SOURCE was never read, or charging a source against the task's write scope, would be wrong.
|
||||
*/
|
||||
internal fun writeTargetPathStrings(paramRoles: Map<String, ParamRole>, parameters: Map<String, Any>): List<String> =
|
||||
pathStringsForRoles(paramRoles, parameters, setOf(ParamRole.PATH))
|
||||
|
||||
/**
|
||||
* Values of the params declared with one of [roles] or — when the tool declares none of them (e.g.
|
||||
* `shell`, whose param is an [ParamRole.EXEC_COMMAND]) — any string value that looks like a path.
|
||||
*/
|
||||
private fun pathStringsForRoles(
|
||||
paramRoles: Map<String, ParamRole>,
|
||||
parameters: Map<String, Any>,
|
||||
roles: Set<ParamRole>,
|
||||
): List<String> {
|
||||
val declared = paramRoles.filterValues { it in roles }.keys
|
||||
return if (declared.isNotEmpty()) {
|
||||
declared.flatMap { extractParamStrings(parameters[it]) }
|
||||
} else {
|
||||
|
||||
+2
-4
@@ -7,9 +7,9 @@ import com.correx.core.toolintent.ToolCallAssessmentInput
|
||||
import com.correx.core.toolintent.ToolCallRule
|
||||
import com.correx.core.toolintent.maxAction
|
||||
import com.correx.core.tools.contract.ToolCapability
|
||||
import com.correx.core.tools.contract.ToolPath
|
||||
import com.correx.core.validation.model.ValidationIssue
|
||||
import com.correx.core.validation.model.ValidationSeverity
|
||||
import java.nio.file.Path
|
||||
|
||||
/**
|
||||
* Effect-based path containment. Dispatches on FILE_READ / FILE_WRITE. For every
|
||||
@@ -34,9 +34,7 @@ class PathContainmentRule : ToolCallRule {
|
||||
var disposition = RiskAction.PROCEED
|
||||
|
||||
for (raw in candidatePathStrings(input.paramRoles, input.request.parameters)) {
|
||||
val candidate = Path.of(raw)
|
||||
val resolvedInput =
|
||||
if (candidate.isAbsolute) candidate else input.workspace.workspaceRoot.resolve(candidate)
|
||||
val resolvedInput = ToolPath.resolve(raw, input.workspace.workspaceRoot)
|
||||
val resolvedReal = input.probe.resolveReal(resolvedInput)
|
||||
val exists = input.probe.exists(resolvedInput)
|
||||
val privileged = privilegedReal.any { resolvedReal.startsWith(it) }
|
||||
|
||||
+10
-5
@@ -6,7 +6,9 @@ import com.correx.core.toolintent.ToolCallAssessment
|
||||
import com.correx.core.toolintent.ToolCallAssessmentInput
|
||||
import com.correx.core.toolintent.ToolCallRule
|
||||
import com.correx.core.toolintent.maxAction
|
||||
import com.correx.core.tools.contract.ParamRole
|
||||
import com.correx.core.tools.contract.ToolCapability
|
||||
import com.correx.core.tools.contract.ToolPath
|
||||
import com.correx.core.validation.model.ValidationIssue
|
||||
import com.correx.core.validation.model.ValidationSeverity
|
||||
import java.nio.file.Path
|
||||
@@ -27,6 +29,12 @@ class ReadBeforeWriteRule : ToolCallRule {
|
||||
ToolCapability.FILE_WRITE in capabilities
|
||||
|
||||
override fun assess(input: ToolCallAssessmentInput): ToolCallAssessment {
|
||||
// A call that declares a SOURCE_PATH takes its content from a file on disk, not from the
|
||||
// model's memory, so there is nothing for this gate to protect: demanding a file_read of the
|
||||
// source (or of the target being replaced by it) would force the very bytes-through-context
|
||||
// round-trip that such tools exist to avoid — and is unsatisfiable for a binary file.
|
||||
if (input.paramRoles.containsValue(ParamRole.SOURCE_PATH)) return ToolCallAssessment()
|
||||
|
||||
val root = input.workspace.workspaceRoot
|
||||
val readReal = input.session.reads.map { realOf(input, root, it) }.toSet()
|
||||
|
||||
@@ -34,7 +42,7 @@ class ReadBeforeWriteRule : ToolCallRule {
|
||||
val observations = mutableListOf<ToolCallObservation>()
|
||||
var disposition = RiskAction.PROCEED
|
||||
|
||||
for (raw in candidatePathStrings(input.paramRoles, input.request.parameters)) {
|
||||
for (raw in writeTargetPathStrings(input.paramRoles, input.request.parameters)) {
|
||||
val resolvedInput = resolveInput(root, raw)
|
||||
val exists = input.probe.exists(resolvedInput)
|
||||
val read = input.probe.resolveReal(resolvedInput) in readReal
|
||||
@@ -58,10 +66,7 @@ class ReadBeforeWriteRule : ToolCallRule {
|
||||
return ToolCallAssessment(issues = issues, observations = observations, disposition = disposition)
|
||||
}
|
||||
|
||||
private fun resolveInput(root: Path, raw: String): Path {
|
||||
val candidate = Path.of(raw)
|
||||
return if (candidate.isAbsolute) candidate else root.resolve(candidate)
|
||||
}
|
||||
private fun resolveInput(root: Path, raw: String): Path = ToolPath.resolve(raw, root)
|
||||
|
||||
private fun realOf(input: ToolCallAssessmentInput, root: Path, raw: String): Path =
|
||||
input.probe.resolveReal(resolveInput(root, raw))
|
||||
|
||||
+2
-3
@@ -7,9 +7,9 @@ import com.correx.core.toolintent.ToolCallAssessmentInput
|
||||
import com.correx.core.toolintent.ToolCallRule
|
||||
import com.correx.core.toolintent.maxAction
|
||||
import com.correx.core.tools.contract.ToolCapability
|
||||
import com.correx.core.tools.contract.ToolPath
|
||||
import com.correx.core.validation.model.ValidationIssue
|
||||
import com.correx.core.validation.model.ValidationSeverity
|
||||
import java.nio.file.Path
|
||||
|
||||
/**
|
||||
* Reference-must-exist gate (anti-hallucination). Dispatches on FILE_READ: a read of a path that is
|
||||
@@ -37,8 +37,7 @@ class ReferenceExistsRule : ToolCallRule {
|
||||
var disposition = RiskAction.PROCEED
|
||||
|
||||
for (raw in candidatePathStrings(input.paramRoles, input.request.parameters)) {
|
||||
val candidate = Path.of(raw)
|
||||
val resolvedInput = if (candidate.isAbsolute) candidate else root.resolve(candidate)
|
||||
val resolvedInput = ToolPath.resolve(raw, root)
|
||||
val exists = input.probe.exists(resolvedInput)
|
||||
val inWorkspace = input.probe.resolveReal(resolvedInput).startsWith(workspaceReal)
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ import com.correx.core.toolintent.ToolCallAssessmentInput
|
||||
import com.correx.core.toolintent.ToolCallRule
|
||||
import com.correx.core.toolintent.maxAction
|
||||
import com.correx.core.tools.contract.ToolCapability
|
||||
import com.correx.core.tools.contract.ToolPath
|
||||
import com.correx.core.validation.model.ValidationIssue
|
||||
import com.correx.core.validation.model.ValidationSeverity
|
||||
import java.nio.file.Path
|
||||
@@ -35,7 +36,7 @@ class StaleWriteRule : ToolCallRule {
|
||||
val observations = mutableListOf<ToolCallObservation>()
|
||||
var disposition = RiskAction.PROCEED
|
||||
|
||||
for (raw in candidatePathStrings(input.paramRoles, input.request.parameters)) {
|
||||
for (raw in writeTargetPathStrings(input.paramRoles, input.request.parameters)) {
|
||||
val resolved = resolveInput(root, raw)
|
||||
val real = input.probe.resolveReal(resolved)
|
||||
val recorded = hashByReal[real]
|
||||
@@ -60,10 +61,7 @@ class StaleWriteRule : ToolCallRule {
|
||||
return ToolCallAssessment(issues = issues, observations = observations, disposition = disposition)
|
||||
}
|
||||
|
||||
private fun resolveInput(root: Path, raw: String): Path {
|
||||
val candidate = Path.of(raw)
|
||||
return if (candidate.isAbsolute) candidate else root.resolve(candidate)
|
||||
}
|
||||
private fun resolveInput(root: Path, raw: String): Path = ToolPath.resolve(raw, root)
|
||||
|
||||
private fun realOf(input: ToolCallAssessmentInput, root: Path, raw: String): Path =
|
||||
input.probe.resolveReal(resolveInput(root, raw))
|
||||
|
||||
@@ -7,10 +7,10 @@ import com.correx.core.toolintent.ToolCallAssessmentInput
|
||||
import com.correx.core.toolintent.ToolCallRule
|
||||
import com.correx.core.toolintent.maxAction
|
||||
import com.correx.core.tools.contract.ToolCapability
|
||||
import com.correx.core.tools.contract.ToolPath
|
||||
import com.correx.core.validation.model.ValidationIssue
|
||||
import com.correx.core.validation.model.ValidationSeverity
|
||||
import java.nio.file.FileSystems
|
||||
import java.nio.file.Path
|
||||
|
||||
/**
|
||||
* Write-scope adherence. When the session has claimed a task that declared affected_paths, an
|
||||
@@ -37,11 +37,8 @@ class WriteScopeRule : ToolCallRule {
|
||||
val observations = mutableListOf<ToolCallObservation>()
|
||||
var disposition = RiskAction.PROCEED
|
||||
|
||||
for (raw in candidatePathStrings(input.paramRoles, input.request.parameters)) {
|
||||
val candidate = Path.of(raw)
|
||||
val resolvedReal = input.probe.resolveReal(
|
||||
if (candidate.isAbsolute) candidate else input.workspace.workspaceRoot.resolve(candidate),
|
||||
)
|
||||
for (raw in writeTargetPathStrings(input.paramRoles, input.request.parameters)) {
|
||||
val resolvedReal = input.probe.resolveReal(ToolPath.resolve(raw, input.workspace.workspaceRoot))
|
||||
if (!resolvedReal.startsWith(workspaceReal)) continue // out of workspace: not this gate
|
||||
val rel = workspaceReal.relativize(resolvedReal)
|
||||
val inScope = matchers.any { it.matches(rel) }
|
||||
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
package com.correx.core.toolintent
|
||||
|
||||
import com.correx.core.events.events.ToolRequest
|
||||
import com.correx.core.events.risk.RiskAction
|
||||
import com.correx.core.events.types.SessionId
|
||||
import com.correx.core.events.types.StageId
|
||||
import com.correx.core.events.types.ToolInvocationId
|
||||
import com.correx.core.toolintent.rules.PathContainmentRule
|
||||
import com.correx.core.toolintent.rules.ReadBeforeWriteRule
|
||||
import com.correx.core.toolintent.rules.ReferenceExistsRule
|
||||
import com.correx.core.toolintent.rules.WriteScopeRule
|
||||
import com.correx.core.tools.contract.ParamRole
|
||||
import com.correx.core.tools.contract.ToolCapability
|
||||
import java.nio.file.Path
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* Plane-2 must judge the SAME path the tool will operate on (ToolPath), and must judge a call's
|
||||
* source path differently from its write target.
|
||||
*
|
||||
* The `~` cases pin the harness bug from the 2026-08 web-ui runs: a correct diagnosis of
|
||||
* `~/.gradle/init.d/offline.gradle` was resolved to `<workspace>/~/.gradle/…`, so the reference gate
|
||||
* called a real file a hallucination and the out-of-workspace prompt never fired.
|
||||
*/
|
||||
class PathNormalizationRuleTest {
|
||||
|
||||
private val workspace = Path.of("/work/project")
|
||||
private val home: String = System.getProperty("user.home")
|
||||
private val tildeTarget = "~/.gradle/init.d/offline.gradle"
|
||||
|
||||
private class FakeProbe(private val existing: Set<Path> = emptySet()) : WorldProbe {
|
||||
override fun exists(path: Path): Boolean = path.toAbsolutePath().normalize() in existing
|
||||
override fun resolveReal(path: Path): Path = path.toAbsolutePath().normalize()
|
||||
}
|
||||
|
||||
private fun input(
|
||||
parameters: Map<String, Any>,
|
||||
capabilities: Set<ToolCapability>,
|
||||
probe: WorldProbe,
|
||||
paramRoles: Map<String, ParamRole> = emptyMap(),
|
||||
reads: Set<String> = emptySet(),
|
||||
tool: String = "file_read",
|
||||
activeTask: ActiveTask? = null,
|
||||
) = ToolCallAssessmentInput(
|
||||
request = ToolRequest(ToolInvocationId("i"), SessionId("s"), StageId("st"), tool, parameters),
|
||||
capabilities = capabilities,
|
||||
workspace = WorkspacePolicy(workspace, emptyList()),
|
||||
probe = probe,
|
||||
paramRoles = paramRoles,
|
||||
session = SessionContext(reads = reads, activeTask = activeTask),
|
||||
)
|
||||
|
||||
@Test
|
||||
fun `a tilde path is not mistaken for a non-existent in-workspace file`() {
|
||||
val real = Path.of(home, ".gradle/init.d/offline.gradle")
|
||||
val r = ReferenceExistsRule().assess(
|
||||
input(
|
||||
mapOf("path" to tildeTarget),
|
||||
setOf(ToolCapability.FILE_READ),
|
||||
FakeProbe(existing = setOf(real)),
|
||||
),
|
||||
)
|
||||
assertEquals(RiskAction.PROCEED, r.disposition)
|
||||
assertTrue(r.issues.isEmpty(), "a real home-directory file must not be reported as a hallucination")
|
||||
assertEquals("true", r.observations.single().facts["exists"])
|
||||
assertEquals("false", r.observations.single().facts["inWorkspace"])
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a tilde path still prompts as an out-of-workspace read`() {
|
||||
// Containment behaviour is preserved: expansion moves the path OUT of the workspace, which is
|
||||
// where it always belonged, so the operator approval fires instead of being silently skipped.
|
||||
val r = PathContainmentRule().assess(
|
||||
input(mapOf("path" to tildeTarget), setOf(ToolCapability.FILE_READ), FakeProbe()),
|
||||
)
|
||||
assertEquals(RiskAction.PROMPT_USER, r.disposition)
|
||||
assertEquals("PATH_OUTSIDE_WORKSPACE", r.issues.single().code)
|
||||
assertEquals(Path.of(home, ".gradle/init.d/offline.gradle").toString(), r.observations.single().facts["resolved"])
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `containment judges a source path as well as a write target`() {
|
||||
val r = PathContainmentRule().assess(
|
||||
input(
|
||||
mapOf("source" to "/etc/hosts", "dest" to "public/hosts"),
|
||||
setOf(ToolCapability.FILE_WRITE),
|
||||
FakeProbe(),
|
||||
paramRoles = mapOf("source" to ParamRole.SOURCE_PATH, "dest" to ParamRole.PATH),
|
||||
tool = "file_copy",
|
||||
),
|
||||
)
|
||||
assertEquals(RiskAction.PROMPT_USER, r.disposition)
|
||||
assertEquals(2, r.observations.size)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `the write-scope gate judges the write target only, not the source`() {
|
||||
// A source path is read, not mutated, so it is not charged against the task's write scope —
|
||||
// otherwise every copy of an in-repo asset would have to widen affected_paths to include it.
|
||||
val r = WriteScopeRule().assess(
|
||||
input(
|
||||
mapOf("source" to "assets/logo.png", "dest" to "public/logo.png"),
|
||||
setOf(ToolCapability.FILE_WRITE),
|
||||
FakeProbe(),
|
||||
paramRoles = mapOf("source" to ParamRole.SOURCE_PATH, "dest" to ParamRole.PATH),
|
||||
tool = "file_copy",
|
||||
activeTask = ActiveTask("42", listOf("public/**")),
|
||||
),
|
||||
)
|
||||
assertEquals(RiskAction.PROCEED, r.disposition)
|
||||
assertEquals(listOf("public/logo.png"), r.observations.map { it.facts["path"] })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a content-from-disk call is exempt from read-before-write even when the dest exists`() {
|
||||
// Overwriting an existing binary via a copy must not demand a file_read of it first — that
|
||||
// read is impossible to satisfy usefully and is exactly the round-trip file_copy removes.
|
||||
val dest = Path.of("/work/project/public/logo.png")
|
||||
val r = ReadBeforeWriteRule().assess(
|
||||
input(
|
||||
mapOf("source" to "assets/logo.png", "dest" to "public/logo.png"),
|
||||
setOf(ToolCapability.FILE_WRITE),
|
||||
FakeProbe(existing = setOf(dest)),
|
||||
paramRoles = mapOf("source" to ParamRole.SOURCE_PATH, "dest" to ParamRole.PATH),
|
||||
tool = "file_copy",
|
||||
),
|
||||
)
|
||||
assertEquals(RiskAction.PROCEED, r.disposition)
|
||||
assertTrue(r.issues.isEmpty())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a model-authored write still requires a prior read`() {
|
||||
val target = "/work/project/src/A.kt"
|
||||
val r = ReadBeforeWriteRule().assess(
|
||||
input(
|
||||
mapOf("path" to target),
|
||||
setOf(ToolCapability.FILE_WRITE),
|
||||
FakeProbe(existing = setOf(Path.of(target))),
|
||||
paramRoles = mapOf("path" to ParamRole.PATH),
|
||||
tool = "file_write",
|
||||
),
|
||||
)
|
||||
assertEquals(RiskAction.BLOCK, r.disposition)
|
||||
assertEquals("READ_BEFORE_WRITE", r.issues.single().code)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user