Reconcile Before You Retry: Recovering Uncertain AI Agent Actions

TL;DR

An AI agent that loses a response cannot safely assume that its action failed. After a timeout, crash, control-plane restore, or worker replacement, the external effect may already exist even when the agent’s own ledger says only prepared, submitted, or unknown.

Recovery therefore needs an action reconciliation boundary between restored workflow state and any new execution attempt. Reconciliation combines the original authorized intent with target-side operation records, current resource state, audit evidence, idempotency records, and the current authorization decision. Its output should be explicit: confirmed executed, confirmed not executed, violated, superseded, or unresolved.

Unknown outcome is a state to investigate. It is not permission to execute again.

Introduction

A production agent is approved to add one Kubernetes role assignment.

The execution worker submits the request.

The connection drops.

No response reaches the worker.

Minutes later, the worker process dies. During incident recovery, the controller and execution ledger are restored from an earlier checkpoint. The action appears unfinished, so the restored workflow considers retrying it.

Meanwhile, the role assignment exists.

Did the original request create it?

Did an administrator create the same assignment manually during the incident?

Did another worker complete the request before being fenced?

Does the target expose an operation identifier that can tie the current object to the original action?

Can the request be repeated safely if that identity is unavailable?

Those questions are not answered by a healthy restored controller.

The previous practical companion on independent recovery admission established that restored state should remain behind an independent recovery-admission boundary. This installment addresses the next question before that boundary can reopen execution: what happened outside the recovered control plane while its knowledge was incomplete?

The answer is reconciliation, not replay.

Recovery Creates an Evidence Gap, Not a Retry Signal

Distributed systems routinely separate request submission from knowledge of the outcome.

A timeout can mean:

• the target never received the request
• the target rejected it
• the target accepted it but the response was lost
• an asynchronous operation began but has not completed
• the operation completed successfully
• the operation completed partially
• another actor changed the same target afterward

Those outcomes have very different recovery implications.

A retry policy that reduces them all to “try again” transfers uncertainty into the target system.

Amazon’s engineering guidance on idempotent APIs describes exactly this class of dilemma. A caller can lose the response even though the requested resource was created. Simply repeating a non-idempotent operation can then create another side effect.

Agentic AI makes the problem more consequential because the workflow may involve identity, network policy, deletion, data movement, backup retention, external communication, or business transactions.

The model should not decide what probably happened.

The runtime needs evidence.

Define Reconciliation as Its Own Workflow

For this series, action reconciliation means determining the supported status of a previously attempted consequential action before deciding whether another attempt is allowed.

It is not synonymous with reading the current target state.

The process has five possible dispositions.

The last state is essential.

Operational systems often pressure engineers to turn uncertainty into a binary result because workflows prefer success or failure.

That pressure should not determine the evidence.

Separate Current State From Causation

Suppose the restored agent intended to set a value to forty-five.

The target currently reports forty-five.

That establishes a current-state fact.

It does not necessarily establish that the agent’s original attempt produced that value.

The same state could have resulted from another controller, an administrator, a recovery procedure, a GitOps reconciler, or a second agent.

The distinction matters whenever the organization needs to establish:

• whether an approval was consumed
• whether another retry is safe
• which identity created the effect
• whether the action occurred before or after revocation
• whether the execution happened inside the approved window
• whether evidence requirements were satisfied
• whether compensation is appropriate

The reconciliation workflow therefore needs more than a desired-state comparison.

A state can exist without attributable causation.

That is one of the most important distinctions in reliable agent execution.

Start With the Original Authorized Intent

Reconciliation begins with the exact action that was approved, not with whatever state the target happens to contain now.

Retrieve the protected execution record:

• action identifier
• authenticated initiator
• agent and executor identities
• tenant and environment
• target identity
• approved operation
• normalized parameters
• policy and approval references
• request or plan digest
• target preconditions
• expected postconditions
• idempotency identifier when supported
• dispatch deadline
• attempt identifiers
• any target operation identifiers already captured

Do not reconstruct these fields from the agent’s conversational memory.

The original action is the baseline against which target evidence is interpreted.

If the protected action record is unavailable, the recovery problem has expanded. The team may still investigate the external system, but it cannot honestly claim to have reconstructed the original authorization from the agent’s narrative alone.

Build an Evidence Hierarchy for Each Target

Different platforms expose different evidence.

Do not force every target into one universal reconciliation algorithm.

Instead, define an evidence profile for each consequential operation.

A useful profile identifies:

1. the strongest available evidence of request acceptance
2. the strongest evidence of operation identity
3. the authoritative read for resulting state
4. evidence of intermediate or asynchronous progress
5. evidence required to establish absence
6. duplicate-handling semantics
7. the observation deadline
8. known ambiguity that cannot be eliminated automatically

For example:

The strongest design uses several of these together when the consequence requires it.

Kubernetes Shows Why Request Evidence and State Evidence Differ

Kubernetes is a useful example because it exposes both object state and API audit information.

Kubernetes objects include a UID that distinguishes one object instance from another object later created under the same name. Resource versions support change tracking and concurrency semantics. Neither value, by itself, identifies the business action that caused every change.

Kubernetes auditing can provide a chronological record of API activity when auditing is configured. Audit events can capture the requesting identity, resource, operation, timestamps, and execution stage. Depending on audit policy, request or response content may also be retained.

That gives a reconciliation process several useful questions:

• Did the intended executor authenticate?
• Did it submit the expected verb against the expected resource?
• Did the API record completion of that request?
• What object UID was involved?
• What does the resource contain now?
• Were there later writes by another identity?
• Is the available audit policy complete enough for the conclusion being requested?

Do not interpret the absence of an audit record as proof that the request never occurred unless the organization’s audit design can support that claim.

The audit subsystem itself can drop or omit information depending on configuration, filtering, buffering, failures, retention, or unavailable historical logs.

Preserve the Audit Stage

A RequestReceived record means the API server saw the request.

It does not mean the request passed authorization, admission, persistence, or downstream processing.

A completed API response provides stronger information about that request path, but the business postcondition may still require another read.

The reconciliation record should therefore preserve which event stage supports its conclusion.

Do not reduce every Kubernetes audit event to operation happened = true.

Idempotency Changes the Retry Decision, Not the Authorization Decision

A target-side idempotency contract can simplify recovery dramatically.

A well-designed service can associate a caller-provided request identifier with the original operation, recognize repeated submissions, reject parameter mismatches, and return a semantically equivalent result for a retry.

That lets a caller recover an earlier result without creating another effect.

Two separate questions remain: is the request still authorized, and can the target safely recognize it as the same logical request?

Idempotency answers the second question.

It does not answer the first.

An approval may have expired while the request was uncertain. A security incident may have suspended the agent. The target resource may have moved into another valid state.

The retry must therefore preserve the original logical request identity and pass current authorization rules.

Do not create a new idempotency identifier simply because the first response disappeared. That turns the retry into a new logical operation.

Do not reuse an old identifier with new parameters either. A changed target, scope, or argument set is a new proposal requiring its own decision.

Classify Retry Safety by Target Contract

The recovery runbook should classify operations before an incident.

Strong Target-Side Idempotency

The service owns a durable request identifier and can return or reconstruct the result of the original logical action.

Recovery preference: Query or retry using the same logical identifier, subject to current authority and the service’s documented contract.

Conditional Mutation

The operation is protected by target-state preconditions such as object identity, resource version, entity tag, generation, or equivalent state constraint.

Recovery preference: Re-read and reconcile. Do not silently refresh the precondition under the old approval.

A failed stale precondition can be valuable evidence that the world changed.

Queryable Asynchronous Operation

The target returns an operation or job identifier whose state can later be queried.

Recovery preference: Resume observation of the original operation rather than create a new one.

Compensation-Based Operation

The side effect may occur only once, but a defined compensation can reverse or reduce its consequences.

Recovery preference: Establish what occurred before deciding whether to compensate. Compensation is a new consequential action with its own authorization and evidence.

No Reliable Duplicate Protection or Operation Lookup

The target cannot reliably identify whether the original action happened.

Recovery preference: Hold, investigate, and use a domain-specific manual or controlled recovery procedure.

This final category is where automation needs to be conservative.

Do Not Use “Resource Already Exists” as Automatic Proof

A retry that receives “already exists” has not necessarily rediscovered its own prior work.

The resource may predate the action.

Another actor may have created it after the first attempt.

The same name may refer to a replacement resource.

Parameters may differ.

This is why mature idempotent interfaces associate caller intent with a stable request identifier instead of inferring duplication merely from similar resource state.

For agent architecture, the rule is:

Matching state can resolve the required postcondition without necessarily resolving action attribution.

Sometimes current-state correctness is enough for the business decision.

Sometimes attribution is required for authorization consumption, incident investigation, billing, or audit.

Define which claim the workflow actually needs.

Preserve Negative Evidence Carefully

Proving that something happened is often easier than proving that it did not.

A resource read that returns “not found” can mean:

• the object does not exist
• the caller lacks permission and the interface obscures existence
• the object was deleted after the original operation
• the query targeted the wrong environment
• the service is returning stale information
• the lookup used the wrong identifier

Similarly, an empty log search can mean no event exists, or that the search does not cover the relevant data.

Before marking confirmed_not_executed, establish what evidence would have existed if the operation had occurred and whether that evidence is expected to remain available.

Where the target cannot support that level of negative proof, keep the outcome unresolved.

That is safer than allowing the workflow’s desire to proceed to manufacture certainty.

Use an Action Reconciliation Record

The recovery process needs a durable artifact representing what has been established.

The following YAML is a proposed record design. It is not a vendor schema and does not authorize a retry.

record_version: "0.1"
record_type: action_reconciliation

action:
  action_id: action-017
  approval_ref: approval-431
  request_digest: sha256:example
  executor_ref: executor-prod
  target_ref: production-resource-52

attempt:
  attempt_id: attempt-017-1
  dispatch_status: submitted
  response_status: lost
  target_operation_ref: null

evidence:
  target_state_ref: observation-942
  target_history_ref: audit-search-117
  idempotency_record_ref: null
  current_authority_ref: authority-check-212

findings:
  current_target_state_matches_intent: true
  original_attempt_attributable: false
  prohibited_side_effect_found: false

disposition:
  status: unresolved
  retry_authorized: false
  dependent_execution: hold

next_action:
  owner: service-owner
  required_evidence:
    - target_operation_attribution
    - or_domain_approved_recovery_decision

The important condition is:

current_target_state_matches_intent: true
original_attempt_attributable: false

Those facts can coexist.

The workflow should not collapse them into success.

Reconciliation Should Not Rewrite the Evidence

Later investigation can improve understanding of an action.

Preserve the earlier uncertainty.

For example:

14:02 attempt submitted
14:03 caller timed out
14:04 disposition = unresolved
14:18 target operation record recovered
14:20 disposition = confirmed_executed

Do not rewrite the 14:04 record to make it look as though the system knew the outcome at that time.

Append the later finding and connect it to the earlier record.

This matters during incident review because it shows which decisions were made under uncertainty and whether the workflow responded correctly to the evidence it actually possessed.

An accurate history is more valuable than an artificially clean history.

Gate Restored Queues Through Reconciliation

Queue recovery is an especially dangerous replay path.

A restored queue may contain:

• work never dispatched
• work dispatched but not acknowledged
• work already completed
• work revoked during the missing interval
• work superseded by another action
• work whose target no longer exists
• work whose original approval expired
• work whose executor has been fenced

Do not treat queue presence as evidence that an action remains executable.

Place restored work into a reconciliation state first.

A recovery queue is not a retry queue until the action has passed this decision path.

Reconcile Before Recovery Admission Reopens Execution

The recovery-admission companion introduced independent recovery admission.

Reconciliation supplies one of the key inputs to that gate.

Before a successor agent resumes consequential work, the recovery decision should know:

• which earlier actions definitely completed
• which definitely did not
• which violated policy
• which were superseded
• which remain uncertain
• which target states were changed manually during the incident
• which approvals were revoked or expired
• which old workers or credentials remain usable
• which idempotency records remain authoritative

An unresolved historical action does not always need to block the entire platform.

Scope the hold to the resources and workflows whose safe next action depends on resolving it.

For example, uncertainty around one firewall policy need not prevent a separately governed read-only knowledge service from operating.

But do not reopen an execution path that can conflict with the unresolved effect.

Define Stop Conditions for Reconciliation

An operational runbook needs explicit reasons to stop investigation-driven automation and escalate.

Hold automatic progression when:

• target identity cannot be established
• target history is incomplete for the required claim
• two authoritative sources disagree
• current state matches intent but causation matters and remains unknown
• idempotency retention may have expired
• approval or revocation history is incomplete
• the target was modified by another actor during the uncertainty window
• the target exposes no safe duplicate-handling contract
• evidence collection itself may have been compromised
• resolving uncertainty would require a new consequential mutation

At that point, manual review is not failure.

It is the designed fallback for a question the automated evidence cannot answer safely.

NIST recovery guidance emphasizes planned, tested recovery procedures rather than improvisation after an event. Action reconciliation belongs in that preparation.

Test the Unknown-Outcome Path Deliberately

Most agent evaluations test obvious success and obvious failure.

The stronger test interrupts the workflow between them.

For one consequential operation:

1. authorize a bounded test action
2. submit it to an isolated or disposable target
3. interrupt the client response path after the target may have received it
4. restart or replace the execution worker
5. withhold its original completion response
6. force the runtime into reconciliation
7. observe whether it discovers the original effect
8. verify whether any retry preserves the original request identity and current authority
9. confirm the intended effect occurs at most once and no prohibited effect occurs
10. preserve the evidence used to reach the disposition

Run variants where the first request:

• never reaches the target
• reaches the target and fails
• succeeds but loses the reply
• begins asynchronous processing
• succeeds and another actor changes the state afterward
• uses an idempotency record that later expires
• produces a state identical to one created by another actor

These cases expose whether the workflow understands uncertainty or merely hides it behind retries.

Measure Reconciliation as an Operating Capability

Do not reduce reconciliation to “eventually succeeded.”

Track measures such as:

Interpret these measures by action class.

A low-value notification workflow and a privileged identity-change workflow should not have the same acceptable uncertainty.

Do not incentivize teams to reduce unresolved counts by prematurely calling actions successful.

Build Target Adapters Around Reconciliation, Not Only Execution

Agent tool design often begins with:

execute(action) -> result

Consequential tools need a richer operating contract.

Conceptually:

prepare(action)      -> proposed change
execute(action)      -> attempt + target reference
observe(action)      -> current target evidence
reconcile(action)    -> supported disposition
recover(action)      -> separately authorized recovery option

Not every platform exposes those methods natively.

A domain adapter can implement the required behavior by combining supported APIs, operation lookups, audit records, and target observations.

The adapter should document what it cannot establish.

That negative information is important.

A tool that says “I can tell whether the state exists, but I cannot prove whether this attempt created it” is more trustworthy than a generic success: true.

Keep Reconciliation Independent From the Model

AI can assist with investigation.

It can summarize logs, compare records, identify inconsistencies, suggest likely causal chains, and explain unfamiliar target data.

Those are useful capabilities.

The final machine-enforced disposition should still depend on evidence and rules appropriate to the operation.

Do not allow the agent that wants to retry to resolve its own uncertainty by writing:

“The operation probably failed, so retrying is safe.”

That is exactly the recursive trust condition this series has been trying to eliminate.

Use AI to analyze the evidence.

Do not use confidence as a substitute for missing evidence.

Assign Ownership Before the Incident

Reconciliation spans several operational domains.

Without this ownership, unknown outcomes become everyone’s problem and therefore no one’s decision.

Document which team has authority to say:

• retry
• do not retry
• compensate
• accept current state
• investigate further
• abandon automation and use manual recovery

The model does not own that decision.

Implement Reconciliation in Phases

Do not start by building a universal cross-platform reconciliation engine.

Start With One Consequential Action

Choose an operation with:

• a known target
• explicit approval
• clear postconditions
• a realistic timeout path
• available target-side evidence

Map every source of evidence.

Define the Evidence Contract

Specify what supports:

• request received
• request accepted
• operation completed
• effect exists
• effect attributable
• effect absent
• safe retry

If the target cannot support one of those claims, say so.

Add the Uncertain Outcome State

Ensure the workflow can persist unresolved without immediately scheduling another attempt.

This sounds simple. In many automation platforms it requires undoing assumptions that every task eventually becomes success or failure.

Exercise Response Loss

Do not merely mock a timeout before the request reaches the server.

Interrupt the response after the target may have committed.

That is the scenario that exposes dangerous retries.

Add Recovery and Queue Restore

Restore the controller or queue while the target retains the original effect.

Confirm that restored work enters reconciliation rather than execution.

Expand by Action Class

Network changes, identity grants, data exports, message delivery, cloud provisioning, and deletion have different evidence and recovery properties.

Standardize the reconciliation interface.

Do not pretend the target semantics are identical.

Conclusion

Agent recovery becomes dangerous when missing knowledge is treated as missing effect.

A timeout, restart, worker replacement, or database restore can separate the workflow’s history from the target’s history. Reconciliation reconnects them before the next consequential action is chosen.

The important distinctions are operational:

Current state is not always causation.

A retry is not always a new action.

Idempotency is not authorization.

Absence of evidence is not always evidence of absence.

And unresolved is a legitimate production state when the system cannot yet support a stronger claim.

Independent recovery admission prevents restored components from reviving stale authority. Action reconciliation prevents those same components from blindly replaying work whose external effects they no longer understand.

The control plane must not grade itself. The retry loop must not fill in missing history either.

Pick one action whose response can be lost after commitment. What target-side evidence would let the recovered agent distinguish “try again” from “it already happened”?

External References

Amazon Builders’ Library: Making retries safe with idempotent APIs

Kubernetes: Kubernetes API Concepts

Kubernetes: Auditing

NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile

NIST SP 800-184: Guide for Cybersecurity Event Recovery

The post Reconcile Before You Retry: Recovering Uncertain AI Agent Actions appeared first on Digital Thought Disruption.