TL;DR
An agent action is not complete when the MCP server returns a successful response. It is complete only when the runtime proves that the intended post-condition exists in the authoritative system and that no unapproved side effect occurred.
The execution runtime should:
revalidate identity, policy, approval, target state, and tool version immediately before execution
acquire a bounded execution lease
generate and enforce a trusted idempotency key
isolate code, processes, files, credentials, and network access
classify timeouts and unknown outcomes before retrying
verify post-conditions through an authoritative read path
record evidence for the action and every recovery step
distinguish retry, rollback, compensation, forward recovery, containment, and manual intervention
stop at explicit points of no return unless the action has stronger approval and recovery planning
The governing principle is:
The runtime does not trust that an action succeeded. It proves what changed, decides whether the outcome is acceptable, and contains the workflow when certainty is not available.
Introduction
Part 1 of this series placed MCP beneath an enterprise agent control plane. Part 2 defined the trusted controller that owns identity, workflow state, policy, approvals, budgets, retries, and termination. Part 3 governed the MCP gateway and server layer through admission control, dynamic tool exposure, schema enforcement, credential separation, routing, and domain authorization.
Part 4 follows an authorized request beyond the MCP server boundary.
This is the point where an agent stops being an information system and starts becoming an operational actor. A ticket is created. A service is restarted. A firewall rule changes. A customer receives a message. A deployment begins. A refund is issued. Data is deleted.
The model may have selected the right tool. The controller may have authorized the request. The gateway may have routed it correctly. The MCP server may have validated the input and returned a valid result.
None of those conditions alone prove that the business outcome occurred safely.
Distributed systems fail between request and response. A timeout can hide a completed side effect. A retry can duplicate a change. A server can report success before an asynchronous job finishes. A downstream API can acknowledge a request that later fails. A cancellation can stop the client from waiting without stopping the external action. A rollback can overwrite a legitimate concurrent change. A compensation step can fail after the original action succeeded.
The execution runtime must own that uncertainty.
Where Part 4 Fits in the Series
The five-part series separates related concerns so that one protocol or service is not expected to perform every control function.
Part
Control boundary
Primary responsibility
Part 1
Architecture boundary
Separate agent reasoning, controller authority, MCP access, runtime execution, and system-of-record state
Part 2
Trusted controller
Own identity, durable state, policy, approvals, budgets, retries, cancellation, and termination
Part 3
MCP gateway and servers
Govern server admission, tool exposure, schemas, credentials, routing, and domain authorization
Part 4
Execution and recovery runtime
Enforce preconditions, idempotency, isolation, verification, evidence, compensation, and rollback boundaries
Part 5
MCP and A2A architecture
Separate agent-to-tool access from agent-to-agent delegation and collaboration
Part 4 is where authorization becomes controlled side effect execution.
Scope and Assumptions
This article assumes that:
the trusted controller has already authorized an action
any required human approval is valid and scoped to the exact action
the MCP gateway has resolved an approved server and tool contract
the MCP server still performs domain authorization and business validation
the enterprise system remains authoritative for the resource state
the runtime can persist execution and recovery evidence
the workflow may span synchronous and asynchronous operations
The article does not assume that every tool supports native rollback or idempotency.
When the downstream capability lacks those properties, the runtime must compensate through stronger preconditions, narrower authority, external deduplication, reconciliation, staged execution, or manual recovery.
A Successful Tool Call Is Not the Success Condition
MCP gives clients and servers structured ways to invoke tools, return results, report errors, track progress, cancel requests, and, in the stable November 25, 2025 specification, represent some long-running work as experimental Tasks.
These protocol outcomes are useful, but they describe the interaction with the MCP server. The enterprise workflow needs a separate success definition.
Consider a tool that requests a rolling restart of a production service.
A successful MCP response might mean:
the request was valid
the server accepted the operation
the downstream API returned an operation identifier
the server created an asynchronous job
The business success condition might require:
only the approved service was targeted
no more than one instance was unavailable at a time
the deployment version remained unchanged
service health returned within the approved window
error rate did not exceed the threshold
no duplicate restart job was created
the system reached a stable final state
Those conditions cannot be inferred from a generic success flag.
The controller should define the expected post-condition before execution, and the runtime should test it afterward.
The Execution Runtime at a Glance
The runtime should treat every consequential tool invocation as a stateful operation with preconditions, an execution envelope, a result classification, verification, and a recovery decision.
The runtime should not allow the model to skip any of these stages by claiming that an action is safe, reversible, already approved, or complete.
The Trusted Execution Envelope
The controller should hand the runtime a versioned execution envelope. The envelope binds the authorized intent to the exact technical action.
A practical envelope contains:
run ID
action ID
authenticated requester
agent workload identity
tenant and organization
target environment
target resource
approved operation
normalized arguments or argument hash
MCP server identity and version
MCP tool identity and contract hash
policy version
approval identifier and expiration
authority budget
idempotency key
preconditions
expected post-conditions
timeout and deadline
retry policy
cancellation behavior
compensation reference
evidence requirements
The runtime must reject the envelope when material values no longer match the current system.
Examples include:
the approval expired
the tool schema changed
the server version changed
the target resource version changed
the environment entered a freeze window
the user or workload identity was revoked
the policy version is no longer compatible
the authority budget was consumed by another branch
the requested operation no longer matches the original plan
An approval is not permission to execute any later variation of the action.
Revalidate Preconditions Immediately Before Execution
Planning and approval can occur seconds or hours before execution. The target system can change during that time.
The runtime should perform a fresh authoritative read immediately before a consequential write.
Useful preconditions include:
target exists
target belongs to the expected tenant
current resource version matches the reviewed version
current state allows the operation
no conflicting job is active
maintenance or change window is open
dependent systems are healthy
capacity is available
backup or restore point exists when required
rollback artifact is still valid
policy and approval are still current
Use optimistic concurrency fields such as record versions, entity tags, generation numbers, or revision IDs where the downstream system supports them.
A precondition failure should normally return the workflow to planning or approval. The runtime should not silently update the action to fit the new state.
That would turn a previously approved action into a different unapproved action.
Idempotency Is a Contract, Not a Retry Flag
Retries are unavoidable in distributed systems. Duplicate side effects are not.
An idempotent operation allows the same authorized intent to be submitted more than once without creating additional mutation. The implementation typically relies on a client-provided idempotency token or an equivalent deduplication record.
The runtime should generate the key, not the model.
A useful idempotency identity binds to:
tenant
workflow type
run ID
action ID
target resource
normalized operation
normalized argument hash
approval version
The server should persist enough information to distinguish:
the first accepted request
a legitimate retry of the same request
a conflicting request that reused the key with different arguments
an expired key outside the allowed replay window
Do not use timestamps as idempotency keys
Timestamps do not reliably identify business intent. Clock skew, parallel requests, and retries from different workers can produce collisions or false uniqueness.
Do not generate a new key for every retry
A new key converts a retry into a new side effect request.
Do not assume an idempotent label proves behavior
MCP tool annotations are advisory unless the server is trusted, and even a trusted server may implement the behavior incorrectly. Idempotency needs tests against the real downstream effect.
Test the unknown-outcome path
The most important test is not two immediate identical calls.
It is:
submit the action
interrupt the response path after the downstream system receives it
retry using the same idempotency key
prove that only one side effect exists
prove that the original result can be recovered
Use Execution Leases to Control Concurrency
A workflow can be delivered more than once by a queue, resumed by a second worker, or retried after a timeout while the original operation is still running.
The runtime should acquire a lease before performing consequential work.
A lease should identify:
action ID
owning worker
acquisition time
expiration time
heartbeat or renewal time
target resource
operation class
Leases prevent two runtime workers from advancing the same action concurrently.
They do not automatically prevent two unrelated workflows from changing the same resource. For that case, use a resource-level lock, optimistic concurrency check, domain conflict rule, or downstream transaction mechanism.
A lease expiration should not immediately trigger another execution. The runtime must first reconcile whether the previous worker produced a side effect.
Isolate the Execution Environment
The execution runtime should receive only the access needed for one bounded action.
For code, shell, file, browser, data-processing, or deployment tools, use controls such as:
ephemeral sandbox or job
non-root execution
read-only base file system
explicit writable working directory
restricted mounted paths
no inherited developer credentials
no container-engine socket
outbound network deny by default
approved destination allowlist
CPU, memory, process, and storage limits
maximum output size
execution deadline
secret injection immediately before use
secret removal after completion
artifact scanning before promotion
cleanup after termination
Isolation should be selected by consequence, not by convenience.
A read-only service status query may need only a narrow API identity. A generated script that changes infrastructure should run inside a disposable sandbox with no ambient authority and should invoke a separately governed deployment interface.
The model should not receive a general shell simply because the desired outcome is difficult to express through a narrow tool.
Separate Prepare, Commit, and Verify When Possible
High-impact tools are easier to govern when execution is divided into phases.
Prepare
The server validates inputs, reads current state, calculates the proposed change, checks dependencies, and returns a change plan or preview.
Commit
The runtime submits the exact approved plan using the same target version, plan hash, tool version, and approval context.
Verify
The runtime queries the authoritative system and tests the defined post-conditions.
This pattern reduces the gap between what the reviewer saw and what the tool executed.
The prepare response should contain a stable plan identifier or hash. The commit operation should reject the request if the underlying resource state or plan inputs changed.
Not every platform supports a native two-phase operation. A domain-specific MCP server can still implement an application-level equivalent by separating simulation, proposal, execution, and status tools.
Timeouts Limit Waiting, Not Necessarily Work
A timeout answers one question:
How long will this caller wait?
It does not automatically answer:
Did the external operation stop?
The stable MCP lifecycle specification recommends configurable request timeouts and cancellation after a request exceeds its timeout. MCP cancellation is intentionally limited. A receiver may ignore a cancellation when the request is unknown, already complete, or cannot be canceled.
This distinction matters operationally.
When a timeout occurs, the runtime should classify the action as an unknown outcome until it can determine whether the side effect started or completed.
Do not immediately retry a timed-out write.
Use one of these reconciliation methods:
query the operation ID
query by idempotency key
read the target resource version
search an authoritative audit log
check the downstream job queue
query the domain-specific status tool
ask an operator when the system provides no reliable status path
A progress notification can show that work is still active, but progress should never extend execution indefinitely. Enforce an absolute deadline even when progress continues.
MCP Tasks Help Track Work, but the Runtime Still Owns Outcome
The stable November 25, 2025 MCP specification describes Tasks as experimental durable state machines for deferred request execution. The protocol includes task status, result retrieval, progress, authorization-context requirements, expiration, and cancellation behavior.
Those capabilities help a runtime track long-running MCP operations.
They do not replace the runtime’s business execution ledger.
An MCP Task can report completed while the enterprise post-condition remains false. A Task can report canceled while the underlying work continues. The stable specification explicitly keeps a canceled task in the canceled state even if execution later completes or fails.
That means the controller and runtime still need to record:
what business action the task represented
which target and approval it was bound to
whether the action produced a side effect
whether the post-condition was verified
whether compensation is required
whether the task record expired before evidence was retained
Current MCP work is also moving the experimental Tasks design toward an extension model. Enterprises should isolate their business workflow state from protocol-version changes and treat MCP task identifiers as one execution reference within a broader run.
Classify the Outcome Before Choosing Recovery
The runtime should not reduce every failure to success or error.
Use a richer result model.
Result class
Meaning
Default runtime response
Not started
Validation or authorization failed before any side effect
Record denial or failure, do not compensate
Accepted
Downstream system accepted asynchronous work
Track operation, do not mark workflow complete
Completed and verified
Expected post-condition exists and no prohibited change was found
Commit success and advance workflow
Completed but unverified
Server reports success, but authoritative verification is unavailable
Pause, retry verification, or escalate
Partially completed
Some approved steps occurred, others did not
Contain and evaluate compensation or forward recovery
Failed before side effect
No mutation occurred
Retry only when the failure is transient and policy allows
Failed after side effect
Mutation occurred but later processing failed
Verify current state and choose recovery
Unknown outcome
Response path failed and side-effect state is uncertain
Reconcile before retry
Compensated
Recovery action restored an acceptable business state
Close as recovered, not as original success
Compensation failed
Recovery action did not reach a safe state
Escalate with evidence and block further automation
This classification should come from trusted runtime and domain logic, not the model’s interpretation of a message.
Verify Against the Authoritative System
Verification is a separate operation with a separate question:
Does the system of record now satisfy the approved post-condition?
The verification path should be independent from the write path where practical.
Examples include:
use a read tool after a write tool
query the enterprise API directly through a read-only identity
inspect the asynchronous operation record
compare the current resource revision with the expected revision
validate health through the monitoring system rather than the deployment system
confirm an external message through the communication platform’s delivery state
verify a financial operation through the ledger rather than the submission API
The verification identity should normally be read-only and should not be able to repair the result silently.
Define expected, tolerated, and prohibited outcomes
Verification should compare more than one value.
For a service restart:
expected: all instances healthy and operation complete
tolerated: one instance still warming within the approved period
prohibited: version changed, capacity reduced, or unrelated service restarted
For a firewall change:
expected: exact rule exists in the approved policy section
tolerated: policy compilation still pending for a bounded period
prohibited: broader source, destination, port, or scope than approved
For a customer message:
expected: one message created for the intended recipient using approved content
tolerated: delivery pending
prohibited: duplicate message, wrong recipient, or unapproved attachment
Verification can also fail
A verification API may be unavailable, stale, or eventually consistent.
The runtime needs a verification policy containing:
maximum verification duration
polling interval
expected consistency delay
independent data sources
required confidence level
escalation threshold
behavior when evidence conflicts
Do not convert lack of evidence into evidence of success.
Practical Post-Condition Examples
Action
Precondition
Authoritative post-condition
Recovery option
Create service ticket
No existing ticket for the idempotency key
One ticket exists with expected fields and owner
Close duplicate or correct bounded fields
Restart service
No deployment or restart already active
Approved instances are healthy and version unchanged
Restore capacity, restart failed instance, or escalate
Change firewall rule
Current policy revision matches reviewed revision
Exact rule exists and effective policy matches approved scope
Remove exact rule or apply compensating policy revision
Send external email
Recipient, content hash, and approval are current
One message record exists for the intended recipient
Send correction or recall when supported
Issue refund
Transaction is eligible and not already refunded
Ledger contains one refund for approved amount
Corrective debit or finance-led compensation
Delete record
Retention, legal hold, and backup checks pass
Record is absent and deletion evidence exists
Restore from protected copy when policy permits
Deploy application
Artifact digest and target revision match approval
Approved digest is running and service objectives pass
Redeploy prior approved artifact or forward-fix
The recovery option is not automatically safe. It requires its own policy, authority, idempotency, verification, and evidence.
Rollback, Compensation, and Forward Recovery Are Different
Teams often use rollback as a generic word for every recovery action. That hides important differences.
Retry
Repeat the same forward operation because the failure is classified as transient and the action is idempotent.
Native rollback
Use a platform mechanism to restore a prior version or transaction state. Examples include redeploying a previous artifact, restoring a configuration revision, or rolling back a database transaction that has not committed.
Compensation
Perform a new business action that offsets an earlier successful action. Cancel a reservation, issue a correcting transaction, remove a created access grant, or send a corrective message.
Compensation does not necessarily restore the exact original state. Concurrent activity or legal and business rules may make that impossible or undesirable.
Forward recovery
Move the system from the failed intermediate state to a valid target state without undoing completed work. This may be safer than reversal when the original action is partially complete.
Containment
Stop further damage while preserving the current state for investigation. Disable the agent, revoke the credential, block the route, isolate the workload, or pause the workflow.
Manual recovery
Transfer control to a qualified operator with the evidence needed to complete or reverse the work safely.
The controller should store the chosen recovery type explicitly. A compensated workflow is not the same as a workflow that never failed.
Choose Recovery Through a Deterministic Decision Path
The model may recommend a recovery path. Trusted policy and domain logic must select and authorize it.
Define Points of No Return
Some operations cannot be safely or meaningfully undone.
Examples include:
releasing sensitive information to an external party
sending a legally binding communication
executing a market transaction
deleting data after the recovery window closes
rotating a credential that immediately invalidates active systems
triggering a physical process
publishing a public statement
The workflow should identify these points before execution.
Before a point of no return, require stronger evidence that:
all reversible validation steps have completed
the target and payload are exact
authorization and approval remain valid
dependencies are healthy
the expected outcome is measurable
the incident and escalation owner is known
compensation limitations are visible to the approver
Place irreversible steps as late as possible in the workflow.
A rollback plan that says “undo the action” is not a recovery plan when the action is irreversible.
Compensation Is Another Production Workflow
A compensating transaction can fail, be retried, conflict with concurrent changes, or create its own side effects.
Treat compensation as a first-class workflow with:
its own action ID
authorization and approval policy
idempotency key
preconditions
bounded credential
execution lease
timeout
result classification
post-condition verification
evidence
escalation path
The runtime must retain the information needed to compensate each completed step. That may include:
previous resource version
created record identifier
original value
reservation identifier
financial transaction identifier
prior artifact digest
approved removal rule
downstream operation ID
Do not attempt to reconstruct recovery information from the model transcript after the failure.
Cancellation Must Reconcile External State
A cancellation request should stop new scheduling and signal in-flight work, but the runtime must not mark the business action canceled until it understands the external state.
A safe cancellation sequence is:
record who requested cancellation and why
stop creating new model and tool operations
send protocol or runtime cancellation signals
revoke the execution lease when appropriate
prevent additional side effects
query the downstream operation state
verify the authoritative resource
compensate or contain partial work when required
record the final disposition
A user closing a window or a client abandoning a request is not proof that the action stopped.
This is especially important for asynchronous jobs and task-based execution.
Build an Execution Ledger
The runtime should preserve an append-only execution ledger that can reconstruct the action and its recovery.
Useful records include:
run and action identifiers
parent workflow state
requester and workload identity
target and environment
server, tool, and contract versions
policy and approval versions
normalized argument hash
idempotency key
lease history
precondition snapshot
start and end times
MCP request and task identifiers
downstream operation identifier
progress events
timeout and cancellation events
structured result classification
verification queries and findings
compensation actions
final post-condition
operator interventions
terminal reason code
The ledger should not contain raw secrets, tokens, or unrestricted sensitive content.
Evidence payloads may need a restricted forensic store separate from ordinary operational telemetry.
Trace the Complete Action Path
OpenTelemetry provides a vendor-neutral foundation for traces, metrics, and logs. Current GenAI conventions can describe agent, model, and tool operations, but the enterprise still needs a runtime telemetry contract for policy, execution, verification, and recovery.
A useful trace contains spans or events for:
controller authorization
runtime preflight
lease acquisition
precondition read
MCP tool execution
downstream operation
progress or task polling
verification
retry decision
compensation
final workflow transition
Record low-cardinality attributes such as:
agent name and version
workflow type
tool name
server version
environment
result class
recovery type
error category
verification outcome
Keep prompts, tool arguments, tool results, customer data, and secrets disabled by default. Capture them only through explicit policy in a protected evidence store.
The goal is not to log everything.
The goal is to preserve enough evidence to prove what was authorized, what executed, what changed, and how the final state was determined.
Example Runtime Policy
The following YAML is an application-level policy example, not an MCP protocol object. It shows how the controller and runtime can define execution, verification, and recovery requirements around a consequential tool.
agent_execution_policy:
version: 5
action:
type: service.restart.execute
risk_tier: high
environment: production
preflight:
revalidate_identity: true
revalidate_policy: true
revalidate_approval: true
require_current_resource_version: true
require_no_conflicting_operation: true
acquire_execution_lease: true
reserve_authority_budget: true
execution:
sandbox_profile: api_only
credential_source: workload_identity
require_idempotency_key: true
maximum_attempts: 2
request_timeout_seconds: 30
absolute_deadline_seconds: 300
maximum_concurrent_actions: 1
retry_unknown_outcome: false
progress:
accept_protocol_progress: true
progress_may_extend_request_timeout: true
progress_may_extend_absolute_deadline: false
verification:
required: true
authoritative_read_tool: service.health.read
maximum_duration_seconds: 180
poll_interval_seconds: 10
expected:
operation_state: completed
healthy_instances: all
application_version: unchanged
prohibited:
duplicate_operation: true
unrelated_service_change: true
recovery:
transient_failure: retry_same_idempotency_key
unknown_outcome: reconcile
partial_completion: contain_then_evaluate
verification_failure: manual_or_compensate
compensation_workflow: service.restart.recover.v2
irreversible_step: false
evidence:
record_precondition_snapshot: true
record_tool_contract_hash: true
record_downstream_operation_id: true
record_verification_result: true
record_recovery_actions: true
capture_raw_credentials: false
capture_sensitive_content: false
emergency:
action_kill_switch: true
tool_kill_switch: true
credential_revocation: true
operator_handoff: true
The values must be changed for the actual tool, platform, service objectives, and recovery capability.
Successful enforcement means a timeout cannot trigger a blind duplicate restart, a stale approval cannot execute, a changed application version fails verification, and operators receive enough evidence to determine the final state.
Example Deterministic Execution Function
The following Python-like example shows the runtime order. It is intentionally framework-neutral.
from dataclasses import dataclass
from enum import Enum
from typing import Any
class Outcome(str, Enum):
VERIFIED = “verified”
NOT_STARTED = “not_started”
UNKNOWN = “unknown”
PARTIAL = “partial”
FAILED = “failed”
@dataclass(frozen=True)
class ExecutionEnvelope:
run_id: str
action_id: str
tool_name: str
arguments: dict[str, Any]
idempotency_key: str
approval_id: str
expected_postcondition: dict[str, Any]
def execute_action(envelope: ExecutionEnvelope) -> Outcome:
runtime_policy.revalidate(envelope)
with execution_leases.acquire(envelope.action_id) as lease:
current = authoritative_reader.read_target(envelope)
preconditions.assert_valid(envelope, current)
prior = idempotency_store.lookup(envelope.idempotency_key)
if prior is not None:
return reconcile_existing(envelope, prior)
idempotency_store.reserve(
key=envelope.idempotency_key,
action_id=envelope.action_id,
argument_hash=hash_arguments(envelope.arguments),
)
evidence.record_preflight(envelope, current, lease)
try:
result = mcp_executor.call_tool(
name=envelope.tool_name,
arguments=envelope.arguments,
idempotency_key=envelope.idempotency_key,
)
except RequestTimeout as exc:
evidence.record_timeout(envelope, exc)
return reconcile_unknown_outcome(envelope)
except TransientFailure as exc:
if retry_policy.permits(envelope, exc):
return retry_same_intent(envelope)
evidence.record_failure(envelope, exc)
return Outcome.FAILED
classified = result_classifier.classify(result)
evidence.record_execution_result(envelope, result, classified)
if classified.requires_reconciliation:
return reconcile_unknown_outcome(envelope)
verification = verifier.check(
envelope=envelope,
observed=authoritative_reader.read_target(envelope),
)
evidence.record_verification(envelope, verification)
if verification.passed:
idempotency_store.complete(
envelope.idempotency_key,
verification,
)
return Outcome.VERIFIED
recovery = recovery_policy.choose(envelope, verification)
recovery_runtime.execute(recovery)
return Outcome.PARTIAL
A production implementation must also handle cancellation, lease expiry, asynchronous task polling, approval revocation, policy changes, circuit breakers, secure credential injection, telemetry, and compensation verification.
The important property is that the model does not decide whether the operation started, whether a timeout is retryable, whether the post-condition passed, or whether compensation is authorized.
Common Execution and Recovery Failure Modes
Failure mode
What happens
Corrective control
Success equals HTTP or MCP success
Workflow advances before the business outcome exists
Define and verify authoritative post-conditions
Blind retry after timeout
Duplicate side effects occur
Classify as unknown outcome and reconcile first
New idempotency key on retry
Every retry becomes a new action
Reuse one trusted key for the same authorized intent
Model-generated idempotency key
Duplicate detection depends on probabilistic output
Generate keys in trusted runtime code
No precondition recheck
Approved action executes against stale state
Read authoritative state immediately before commit
Cancellation equals stopped
Client exits while external work continues
Reconcile downstream state before terminal cancellation
Compensation treated as simple undo
Recovery overwrites concurrent legitimate changes
Use domain-specific compensating actions and verification
Rollback plan stored only in prompt
Required recovery data is missing during failure
Persist recovery metadata in the execution ledger
Verification uses the write response
Server validates its own claim of success
Use an independent authoritative read path
Progress extends forever
Misbehaving work consumes resources indefinitely
Enforce an absolute deadline
Shared long-lived credential
One action has broad reusable authority
Inject short-lived action-scoped credentials
Sandbox has ambient access
Generated code reaches files, networks, or sockets outside scope
Use explicit mounts, egress deny, and ephemeral execution
Recovery has no idempotency
Compensation repeats and creates new damage
Treat recovery as a first-class idempotent workflow
Irreversible action occurs early
Later validation failure cannot be recovered
Place points of no return after reversible checks
Evidence records only final status
Operators cannot reconstruct partial work
Store preflight, execution, verification, and recovery events
A Practical Implementation Sequence
Select one bounded action
Begin with one action that has clear preconditions, a measurable post-condition, and a realistic recovery path.
Good starting actions include creating a draft, opening a ticket, adding a bounded label, or restarting one noncritical service instance.
Do not begin with arbitrary shell execution or broad production changes.
Define the action contract
Document:
target type
allowed arguments
expected side effects
preconditions
post-conditions
prohibited outcomes
idempotency scope
timeout
retry classification
cancellation behavior
recovery options
point of no return
evidence requirements
Add an execution ledger
Persist one action record before invoking the tool. Store state transitions and evidence as the operation advances.
Implement preflight reads
Query the authoritative system, compare versions, check conflicts, and reserve the authority budget.
Add idempotency and leases
Prove that duplicate queue delivery, worker restart, and response loss do not create duplicate side effects.
Isolate the runtime
Remove ambient credentials and network access. Inject only the short-lived identity required for the action.
Add result classification
Distinguish accepted, completed, partial, failed-before-side-effect, failed-after-side-effect, and unknown outcomes.
Implement independent verification
Create a read path that does not depend on the model or the write response.
Implement one recovery workflow
Test a native rollback, compensating action, forward fix, or manual handoff using the same control discipline as the original action.
Test cancellation and timeouts
Interrupt execution at every boundary:
before downstream receipt
after downstream receipt but before response
during asynchronous processing
after side effect but before verification
during compensation
Add observability and alerts
Alert on unknown outcomes, verification failures, compensation failures, lease expiry, duplicate suppression, and irreversible actions requiring manual review.
Expand authority gradually
Move through read, simulate, propose, approved execution, and bounded autonomy only after evidence shows that the runtime controls failure correctly.
Runtime Validation Tests
A production readiness test suite should include:
duplicate delivery with the same idempotency key
duplicate delivery with conflicting arguments
timeout before downstream receipt
timeout after downstream receipt
lost response after successful mutation
stale resource version
expired approval
revoked workload identity
changed tool contract
concurrent action against the same target
task record expiration
ignored cancellation
progress notification flood
partial multi-step completion
verification API outage
eventual consistency delay
compensation retry
compensation failure
operator kill switch
forensic reconstruction from evidence
A workflow that passes only the happy path is not production-ready.
Operating Metrics for Agent Execution
Measure the runtime as a production service.
Useful metrics include:
authorized actions started
precondition failures
duplicate actions suppressed
idempotency conflicts
execution lease conflicts
timeouts by action type
unknown-outcome rate
verification latency
verification failure rate
retries by classified cause
cancellation-to-containment latency
partial completion rate
compensation attempts
compensation success rate
manual recovery rate
irreversible action count
sandbox policy violations
post-condition regressions
cost per verified action
The most important denominator is often verified business outcomes, not tool calls.
A low tool error rate can hide a high verification failure rate.
Operational Ownership
Capability
Primary owner
Evidence required
Action contract
Business service owner and application owner
Allowed outcomes, prohibited outcomes, recovery requirements
Runtime engine
AI platform or workflow platform team
Availability, state recovery, leases, limits, release history
Idempotency
Tool owner and runtime owner
Key scope, retention, conflict tests, duplicate suppression results
Sandbox and execution identity
Platform security and identity teams
Runtime profile, scopes, egress policy, credential lifetime
Preconditions and verification
Enterprise system owner
Authoritative read path, consistency behavior, success criteria
Compensation workflow
Business and application owners
Recovery steps, authority, tests, manual fallback
Telemetry and evidence
SRE, security operations, and compliance
Trace contract, retention, redaction, access controls
Incident response
Agent service owner and operations
Kill switches, escalation path, recovery runbook
Final business outcome
Named agent service owner
Verified result, residual risk, acceptance criteria
The team that owns the MCP server may not own the complete business recovery. That ownership must be assigned before production deployment.
Execution, Verification, and Recovery Checklist
Before execution
Identity, policy, approval, and budget are revalidated.
Server, tool, schema, and contract versions match approval.
Current resource state is read from the authoritative system.
Preconditions and concurrency rules pass.
An execution lease is acquired.
The trusted runtime generates the idempotency key.
The short-lived credential and sandbox are prepared.
The post-condition and recovery policy are already defined.
During execution
The request uses the approved target and normalized arguments.
Timeouts and an absolute deadline are enforced.
Progress is correlated but cannot extend work indefinitely.
Output size and resource use are bounded.
Protocol, tool, and downstream operation identifiers are recorded.
Cancellation stops new work and signals in-flight work.
After execution
The result is classified beyond success or failure.
Unknown outcomes are reconciled before retry.
The authoritative system is queried independently.
Expected, tolerated, and prohibited outcomes are evaluated.
Evidence is retained without exposing raw credentials.
The workflow advances only after verified success.
During recovery
Retry, rollback, compensation, forward recovery, containment, and manual recovery are distinguished.
Recovery has its own authority and idempotency key.
Compensation progress is persisted.
Compensation is independently verified.
Irreversible steps trigger stronger review.
Failed recovery blocks further autonomous action and escalates.
Conclusion
The execution runtime is the last control boundary before an authorized agent proposal becomes a real enterprise side effect.
Its job is not merely to send the MCP request. It must revalidate the action, acquire exclusive execution authority, enforce idempotency, isolate the runtime, limit resources, classify uncertain outcomes, verify the authoritative post-condition, preserve evidence, and choose a safe recovery path when the outcome is wrong or unknown.
This is why protocol success cannot be the business success condition.
A tool response may prove that a server accepted or completed a request. It does not prove that the correct resource changed, that the change stayed inside the approved boundary, that an asynchronous job finished, that no duplicate action occurred, or that the service reached a healthy state.
The practical operating model is:
Execute inside a bounded runtime. Verify against the system of record. Retry only when the intent is idempotent. Compensate through a separately governed workflow. Escalate when the final state cannot be proven.
Part 5 will complete the series by placing MCP and A2A into one enterprise architecture. It will show where agent-to-tool execution ends, where agent-to-agent delegation begins, and how identity, task state, artifacts, policy, and observability should cross that boundary without creating a second uncontrolled control plane.
External References
Model Context Protocol: LifecycleCanonical URL: https://modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle
Model Context Protocol: CancellationCanonical URL: https://modelcontextprotocol.io/specification/2025-11-25/basic/utilities/cancellation
Model Context Protocol: ProgressCanonical URL: https://modelcontextprotocol.io/specification/2025-11-25/basic/utilities/progress
Model Context Protocol: TasksCanonical URL: https://modelcontextprotocol.io/specification/2025-11-25/basic/utilities/tasks
Model Context Protocol: ToolsCanonical URL: https://modelcontextprotocol.io/specification/2025-11-25/server/tools
Model Context Protocol: SEP-2663 Tasks ExtensionCanonical URL: https://modelcontextprotocol.io/seps/2663-tasks-extension
Amazon Web Services: REL04-BP04 Make mutating operations idempotentCanonical URL: https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_prevent_interaction_failure_idempotent.html
Microsoft Azure Architecture Center: Compensating Transaction patternCanonical URL: https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction
OpenTelemetry: Inside the LLM Call: GenAI Observability with OpenTelemetryCanonical URL: https://opentelemetry.io/blog/2026/genai-observability/
How Many AI Workloads Can My GPU Platform Really Support?
TL;DR An expected workload count is not a GPU requirement. Thirty concurrent notebooks, RAG services, inference endpoints, fine-tuning jobs, or distributed training…
Next PostBrownfield vSphere to VMware Cloud Foundation 9.1: Import, Converge, or Rebuild?Introduction The phrase “import an existing vCenter” sounds safer than it really is. It suggests that VMware Cloud Foundation reads an inventory, registers a few objects, and leaves the underlying…
The post How to Execute, Verify, and Roll Back Agent Actions: Enterprise Agent Control Plane Series, Part 4 appeared first on Digital Thought Disruption.