feat(qa): remote NIM provider + headless-QA robustness
Enable autonomous QA through a remote OpenAI-compatible provider (NVIDIA NIM) and harden the tool/approval path so unattended multi-stage runs complete. - inference: add openai_compat provider (Bearer chat-completions for NIM/OpenAI), dispatched by provider type "nim"/"openai"; key via api_key/api_key_env. - server: bind configured [server] host/port instead of a hardcoded 8080; POST /sessions accepts an optional `intent` (WS parity) for intent-driven workflows. - kernel: thread the bound operator profile's approval_mode into per-tool gating so auto/yolo enable unattended approval (engine still consulted; policy/plane-2 BLOCK stays terminal); on a recoverable tool failure feed the tool's arg-schema back into context so the model self-corrects instead of repeating a malformed call. - tools: split deletion out of file_write into a separate, explicitly-named file_delete tool — a model can no longer delete a file by getting a write-mode parameter wrong. - server: add GET /metrics/tool-reliability — per-model tool-call validity from the event log (measurement groundwork for capability-aware routing). - docs: update AGENTS.md across kernel, tools, server, inference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -13,7 +13,8 @@ CORREX kernel team. This is the integration point for all other `core/` modules.
|
||||
- `SessionOrchestrator` / `DefaultSessionOrchestrator` — primary entry point for launching and advancing sessions through workflow stages.
|
||||
- `OrchestrationState` / `OrchestrationReducer` (`DefaultOrchestrationReducer`) / `OrchestrationProjector` / `OrchestrationRepository` — standard event-sourcing stack for orchestration state.
|
||||
- `RetryCoordinator` / `DefaultRetryCoordinator` — manages retry logic per `RetryPolicy`.
|
||||
- `ApprovalGateway` — kernel-side approval bridge; calls `core:approvals` engine before executing gated operations.
|
||||
- On a recoverable tool failure `dispatchToolCalls` feeds the failing tool's argument schema back into context alongside the error (`toolArgsHint`), so the model self-corrects a malformed call instead of repeating it — the contract stays strict; the feedback is what loosens.
|
||||
- `ApprovalGateway` — kernel-side approval bridge; calls `core:approvals` engine before executing gated operations. Per-tool gating in `dispatchToolCalls` builds the `ApprovalContext` mode from the session's bound operator profile (`boundProfile.approvalMode`, mapped by `approvalModeFor`): unset/`prompt` keeps a human in the loop (default), `auto` auto-approves up to T2, `yolo` all tiers, `deny` blocks above T0. The engine is always consulted (Invariant #4 holds); policy/plane-2 BLOCK stays terminal regardless of mode.
|
||||
- `ReplayOrchestrator` / `ReplayInferenceProvider` / `ReplayStrategy` — deterministic replay of a session from its event log. `ReplayInferenceProvider` returns recorded responses — no live LLM (Hard Invariant #8).
|
||||
- `SubagentRunner` / `InSessionSubagentRunner` — runs sub-agent invocations within an active session.
|
||||
- `StaticAnalysisRunner` / `ProcessStaticAnalysisRunner` — runs static analysis tools and records results as events.
|
||||
|
||||
+25
-3
@@ -538,7 +538,8 @@ abstract class SessionOrchestrator(
|
||||
continue
|
||||
}
|
||||
val toolEntries = dispatchToolCalls(sessionId, stageId,
|
||||
inferenceResult.response.toolCalls, stageConfig, effectives)
|
||||
inferenceResult.response.toolCalls, stageConfig, effectives,
|
||||
approvalModeFor(session.state.boundProfile?.approvalMode))
|
||||
val fatalEntry = toolEntries.firstOrNull { it.content.startsWith("FATAL:") }
|
||||
if (fatalEntry != null) {
|
||||
emitProcessResultEvents(sessionId, stageId, stageConfig)
|
||||
@@ -631,12 +632,33 @@ abstract class SessionOrchestrator(
|
||||
}
|
||||
|
||||
@Suppress("CyclomaticComplexMethod")
|
||||
// Maps the operator profile's free-text approval_mode onto the engine's [ApprovalMode].
|
||||
// Unset/unknown falls back to PROMPT so an absent profile keeps the human-in-the-loop
|
||||
// default; only an explicit auto/yolo opts a session into unattended approval.
|
||||
private fun approvalModeFor(profileMode: String?): ApprovalMode =
|
||||
when (profileMode?.trim()?.lowercase()) {
|
||||
"deny" -> ApprovalMode.DENY
|
||||
"auto" -> ApprovalMode.AUTO
|
||||
"yolo" -> ApprovalMode.YOLO
|
||||
else -> ApprovalMode.PROMPT
|
||||
}
|
||||
|
||||
// On a recoverable tool failure, append the tool's argument schema to the error fed back to the
|
||||
// model so it can self-correct its next attempt (it already sees its own malformed call in the
|
||||
// assistant entry above) rather than repeating the same mistake. Kept compact to bound context.
|
||||
private fun toolArgsHint(tool: Tool?): String =
|
||||
tool?.let {
|
||||
"\nThe '${it.name}' tool requires arguments matching this JSON schema — re-issue the " +
|
||||
"call with corrected arguments: ${it.parametersSchema}"
|
||||
}.orEmpty()
|
||||
|
||||
private suspend fun dispatchToolCalls(
|
||||
sessionId: SessionId,
|
||||
stageId: StageId,
|
||||
toolCalls: List<ToolCallRequest>,
|
||||
stageConfig: StageConfig,
|
||||
effectives: RunEffectives,
|
||||
approvalMode: ApprovalMode,
|
||||
): List<ContextEntry> {
|
||||
val executor = effectives.executor ?: return emptyList()
|
||||
val processResultSlots = stageConfig.produces.filter { it.kind.id == "process_result" }
|
||||
@@ -765,7 +787,7 @@ abstract class SessionOrchestrator(
|
||||
val projectId = effectives.policy?.workspaceRoot?.let { ProjectIdentity.of(it.toString()) }
|
||||
val approvalCtx = ApprovalContext(
|
||||
identity = ApprovalScopeIdentity(sessionId, stageId, projectId = projectId),
|
||||
mode = ApprovalMode.PROMPT,
|
||||
mode = approvalMode,
|
||||
)
|
||||
val requestId = ApprovalRequestId(UUID.randomUUID().toString())
|
||||
val toolPreview = computeToolPreview(toolCall.function.name, parameters)
|
||||
@@ -938,7 +960,7 @@ abstract class SessionOrchestrator(
|
||||
?: result.output
|
||||
is ToolResult.Failure -> {
|
||||
if (!result.recoverable) "FATAL: ${result.reason}"
|
||||
else "ERROR: ${result.reason}"
|
||||
else "ERROR: ${result.reason}${toolArgsHint(tool)}"
|
||||
}
|
||||
}
|
||||
val resultEntry = ContextEntry(
|
||||
|
||||
Reference in New Issue
Block a user