Building an AI Agent Execution Ledger That Survives Restarts

TL;DR

An AI agent execution ledger records which approved action a worker has claimed, whether execution was prepared, and what remains unresolved. Its critical operation is a committed state transition that prevents two workers from independently spending the same approval. A process restart must not turn claimed or uncertain work back into available authority.

This companion supplies a single-host SQLite lab with separate worker processes, abrupt process-exit tests, and an old-database recovery counterexample. Twenty-eight local tests passed. The lab makes no model or infrastructure calls and implements no authenticated approval issuer. It demonstrates selected persistence behavior, not exactly-once execution or production security.

A missing completion record is a reason to investigate, not permission to execute again.

Introduction

Two workers receive the same approved change. Both inspect the approval and find it active. Both submit the operation.

The request may be correctly bound to its target, parameters, and validity window. The failure is elsewhere: nothing established which worker was entitled to consume that authority.

Now add a restart. A worker records its intention to dispatch, sends the request, and loses its connection before recording the result. Its replacement sees unfinished work and tries again.

The conditional Kubernetes patch from the previous companion can reject a repeated mutation whose starting-state conditions no longer hold. That is useful target protection. It does not establish who claimed the approval, resolve the lost response, or provide a general retry contract for the next tool.

Binding AI Agent Approvals to Kubernetes Changes defined the relationship between approval and the exact request. This installment implements a narrower part of the surrounding workflow: persistent claim ownership and conservative handling of interrupted work.

The central design decision is deliberate. This lab refuses automatic takeover of a committed claim. That can leave legitimate work stranded until someone reconciles it. The alternative, silently treating uncertainty as unused authority, creates a harder failure to contain.

Define the Ledger’s Job Before Choosing Its Database

The ledger answers questions about the execution workflow. It does not independently establish that the business approved the action or that the target performed it.

Keep four claims separate.

Approval exists means an accepted authority issued a decision for the operation. A worker owns the claim means the workflow assigned that approval’s permitted use. Dispatch was prepared means the system durably recorded its intention to cross the execution boundary. The effect is verified requires evidence from the affected system.

The lab implements the middle two claims. It accepts a separately seeded synthetic approval and deliberately stops short of target execution and verified completion.

The existing DTD trusted agent controller design describes authoritative workflow state as distinct from model context. The practical extension here is that the state must remain meaningful when more than one process uses it and when one process disappears.

Preserve the Previous Request as an Unapproved Artifact

The included example was generated from the preceding Bound Request Builder’s unchanged synthetic snapshot. It retains the same conditional thirty-to-forty-five change to the unconsumed retention-fixture ConfigMap.

The request still says authorization: not_established.

A separate lab function, seed_demo_approval, inserts a synthetic grant that references the exact request bytes. That function stands in for an approval issuer; it does not authenticate an approver or verify a signature.

Do not copy that provisioning mechanism into a production service and expose it to the agent. Registering a proposal and issuing authority remain different operations.

Assign Identifiers Outside the Model

Use distinct identifiers for the logical action, approval, worker, and claim. Bind the approval to the immutable request and permitted executor.

The lab enforces unique approval and logical-action identifiers within its database. That prevents accidental duplicate registration under those identifiers. It does not detect two differently named actions that request the same business effect.

Do not generate a new action identifier merely to escape an existing claim. A successor action needs an explicit relationship to the earlier one and a justified decision about its unresolved effects.

Make Claim Ownership a Committed Transition

A separate read followed by an unconditional write is not a claim protocol.

The operation should acquire the appropriate transaction, inspect current authority, perform a conditional state change, and record the transition before committing. Only then should it return an accepted claim.

SQLite’s transaction documentation states that separate connections and processes can read concurrently, but only one write transaction is active at a time. BEGIN IMMEDIATE attempts to begin that write transaction at entry and can return SQLITE_BUSY when another writer holds it.

For this lab, that provides a straightforward local coordination mechanism. Each worker opens its own connection to the same database.

The conditional update at the center of the operation has this shape:

UPDATE approvals
SET state = 'claimed',
    claim_id = :claim_id,
    worker_ref = :worker_ref,
    claimed_at = :now
WHERE approval_id = :approval_id
  AND state = 'available'
  AND revoked = 0
  AND request_sha256 = :request_sha256
  AND executor_ref = :executor_ref
  AND not_before <= :now
  AND dispatch_before > :now;

This is an explanatory SQL excerpt, not a standalone approval endpoint. The application validates the inputs, opens the transaction, checks that exactly one row changed, inserts the transition event, and commits.

Zero matching rows must not become “probably already approved.” A storage error must not become permission to skip recording.

Sample Time After Waiting for the Transaction

A worker can wait for database access while its approval expires.

The implementation samples its clock after acquiring the write transaction, then checks the validity window. It checks again when advancing from claimed to prepared.

The supplied clock uses synthetic integer ticks. It does not validate real time, synchronization, or resistance to clock manipulation. A production service needs an accepted time source and defined behavior when time cannot be trusted.

The lab’s database lock timeout is also separate from approval validity. Waiting two seconds for a lock does not extend the approval by two seconds.

Commit the Event with the Claim

The current row and its transition event are written in the same local transaction. A test deliberately prevents insertion of the claim event and confirms that the preceding state change rolls back.

That avoids one local inconsistency: a claimed action with no corresponding transition record from the operation that claimed it.

It does not make the event history tamper-proof. A database administrator can modify both tables. Independent evidence custody remains a separate control.

Keep Network Calls Outside This Transaction

Do not hold the database transaction open while calling Kubernetes, a model provider, or another infrastructure service.

That increases contention and still does not make the remote effect part of the local commit. Rolling back the ledger cannot undo an external request already accepted by its target.

The transaction protects the workflow transition. The adapter and target must supply the appropriate execution and duplicate-handling behavior.

Prefer Explicit Uncertainty to Automatic Takeover

The lab supports a small state model:

Revocation is a separate flag. Revoking a prepared action does not erase the prepared record or prove that an external operation was canceled.

The diagram highlights the missing reverse arrow. The normal interface offers no transition from claimed, prepared, or unresolved back to available.

This is stricter than a general-purpose job queue. It is intended to expose the safety cost of treating consequential actions like disposable computation.

An Expired Worker Lease Does Not Prove the Worker Stopped

A worker can pause, lose its heartbeat path, or become disconnected while retaining the ability to reach the target. Reassigning its action may create a second active executor.

A replacement design needs a mechanism that prevents stale workers from acting. This is often described as fencing: the receiving execution boundary rejects work from a superseded owner.

A generation number checked only inside a worker is insufficient. The relevant receiver must enforce it, and every usable execution path must honor the same boundary.

The lab does not implement fencing or automatic takeover. Its response is to preserve the claim and hold further progression.

Lost Claim Replies Need Reconciliation Too

A claim transaction may commit even if its caller never receives the accepted response.

The caller should not create another approval or assume the first claim failed. It needs to inspect the authoritative record and establish whether it owns the recorded claim.

The lab’s repeated claim call does not issue another accepted claim, even to the same worker. An authenticated production service could provide a carefully defined recovery response, but that is different from granting another use.

Run the Single-Host Persistence Exercise

The Execution Ledger Lab v0.1 contains the implementation, synthetic request files, twenty-eight tests, and a demonstration runner. It leaves the earlier packages unchanged.

Use Python 3.10 or later, its standard-library SQLite module, and a trusted local filesystem. Local validation used Python 3.13.5 and SQLite 3.46.1 on Linux. Other environments were not tested.

Do not place the database directly on a shared network filesystem for this exercise. SQLite’s Appropriate Uses guidance warns about direct multi-computer access and unreliable locking in some network filesystems. A future multi-host design needs an appropriate service boundary and separately validated storage architecture.

From the extracted directory:

cd rtb-execution-ledger-lab
umask 077

python -m unittest discover -s tests -v

python run_lab.py 
  --output ../ledger-run-01

The output directory must not exist, and its parent must exist. Preserve partial outputs after a failure and use another directory for a rerun.

The runner spawns local processes. It does not invoke a model, call Kubernetes, or mutate an external system.

Inspect the Persistence Settings

The implementation selects SQLite’s DELETE journal mode and sets synchronous=EXTRA on each operational connection, then checks the active settings.

SQLite documents that EXTRA adds directory synchronization after unlinking the rollback journal in this mode. That is a durability setting, not evidence that this environment has survived power loss.

The local exercises terminate application processes. They do not test host failure, disk failure, storage-controller behavior, or power interruption.

“Survives a worker exit” and “survives the required infrastructure failure domain” remain different acceptance claims.

Examine the Three Demonstrations

Two complete demonstration runs produced the same outcome counts. The winning worker and generated claim identifiers varied.

The final row is intentionally a counterexample. It is not a successful recovery-security test.

The runner’s summary makes that visible:

{
  "crash_recovered_state": "prepared",
  "expected_lab_observations": true,
  "old_database_copy_reaccepted_claim": true,
  "production_authorization": "not_assessed",
  "race_accepted_claims": 1,
  "race_rejected_claims": 7,
  "replacement_claim_accepted": false
}

Exit code 0 means those expected observations occurred, including the unsafe old-copy behavior. Exit code 1 means the observations did not match expectations. Exit code 2 means the run was incomplete.

Do not wire this result directly into production promotion. The counterexample exists to identify a control still required outside the database.

What the Twenty-Eight Tests Cover

The tests include actual process competition, abrupt exit before a claim commit, abrupt exit after preparation, repeated preparation, request-digest mismatch, wrong executor reference, expiry, revocation, lock contention, missing databases, and failure to append the transition event.

They also check that an unapproved input remains unapproved and that unresolved work does not become claimable again.

These are selected tests written alongside the implementation. Passing them does not establish exhaustive schedule coverage or independent adjudication of the design.

All process identities and clock values are trusted test inputs. The package does not authenticate callers, issue signed approvals, or create administrative isolation between workers.

Understand the Crash Window the Database Cannot Close

Consider three points where an execution worker can disappear.

Before the claim commits, the local transaction may leave no accepted claim. The tested uncommitted process exit rolls back the attempted change. This is safe only because the protocol forbids external execution before the required commit.

After the claim commits but before preparation, ownership exists even though no dispatch intent was recorded. The lab holds that ownership. A production recovery procedure may eventually establish that another attempt is permissible, but silence alone does not establish it.

After preparation commits, the worker may have stopped before sending, during transmission, or after the target accepted the request. The same recovered ledger state can describe all three situations.

That is why prepared is not synonymous with “executed” or “not executed.”

Do Not Store the Simulated Target in the Same Transaction and Claim the Problem Is Solved

An exercise can atomically update an approval row and a fake target row in one database. That tests a different architecture from an approval service calling an external infrastructure API.

When the real target is Kubernetes, its persistence boundary remains separate. Keep the simulation honest about that gap.

The execution and verification runtime must join the ledger record with the target’s supported observations, operation records, and duplicate-handling contract. A useful local transaction is not a distributed transaction merely because both systems store data reliably.

Idempotency Belongs at the Effect Boundary Too

Amazon’s Making retries safe with idempotent APIs describes recording request identity together with the service’s mutating work and checking parameter consistency on repeated requests.

A client-side ledger cannot impose that behavior on an arbitrary target.

The conditional Kubernetes request from the preceding companion supplies another form of protection: required UID, revision, and starting-data checks. Kubernetes documents conditional updates as a way to detect conflicting state. Preserve those checks rather than refreshing them under the original approval.

Even when a repeated patch is rejected, investigate the original outcome. Rejection of a second request does not identify which earlier actor produced the current state.

An Older Valid Database Can Restore Spent Authority

The recovery counterexample is the most important result in this lab.

The runner copies the database before a claim. It then consumes the approval in the current database and opens the old copy as though that copy were current. Both databases accept a claim because each sees internally consistent local history.

No corruption is required.

The old copy is missing a later fact. Its transaction machinery cannot discover that fact from the records it contains.

This is the operational reason to treat agent rollback as an authority-restoration problem, not merely a software restart. DTD’s AI agent rollback and recovery patterns provide the broader incident-response companion.

Keep Recovery Admission Outside the Restored History

Before making a restored ledger authoritative, stop dispatch through a separately governed mechanism. Reconcile newer revocations, consumed approvals, queued requests, and target effects. Establish whether stale workers still possess usable execution access.

An external recovery generation can help only if its current value survives the failed boundary and relevant receivers enforce it. Restoring the generation counter from the same old backup simply restores the same ambiguity with another field attached.

Do not let the recovering agent attest that its own approval history is current.

The lab deliberately implements none of these external recovery controls. Its old-copy result is evidence that they are needed before claiming safe restoration.

Recovery Does Not Mean Erasing the Claim

If the operation never happened and another attempt is justified, preserve the original claim and record the recovery decision. A successor action should identify what it replaces and what authority now applies.

Do not directly change the old row to available to clear a dashboard.

A new action also cannot make an old worker harmless. Contain or fence the stale execution path before permitting a conflicting successor.

Keep Revocation Separate from Cancellation

The ledger’s revocation flag blocks subsequent claim or preparation checks when those checks see it. It does not cancel work already accepted by a target.

OWASP’s Transaction Authorization Cheat Sheet recommends a final authorization check tied to execution. The integration must define where that check happens and what guarantee it supports.

This lab checks before preparation. A production adapter still needs current authority at its declared dispatch boundary, along with a defined treatment of the interval before target acceptance.

During a partition, a worker unable to obtain mandatory current authority should hold new consequential work under this proposed operating mode. Continuing under a cached grant is a different design requiring explicit delegation, expiration, and accepted revocation delay.

Do not let a worker choose that degraded mode because it wants to finish.

Treat Database Failures as Control Failures

A busy database means the workflow could not acquire the required transaction within its configured wait. It is not a reason to execute first and write the record later.

Likewise, a missing database is not an empty database. The operational loader refuses to create a new one implicitly. Silent recreation could make every prior claim disappear.

Some conditions require an availability tradeoff. This deliberately conservative implementation can strand unused approvals after a worker failure.

Budget for reconciliation, or implement and validate a stronger recovery protocol. Do not conceal the tradeoff by describing automatic retries as resilience.

Integrate the Ledger Behind a Protected Service

In production, workers should use an authenticated service interface, not receive database write access.

The service must obtain initiating identity, executor identity, approval status, and request binding through governed sources. Keep approval issuance, claim management, revocation, and evidence administration appropriately separated.

The local worker-reference strings are not credentials. The request hash is not a signature. The SQLite event table is not an independently protected audit repository.

A production adapter should consume the protected request associated with the claim, rather than accepting a different agent-supplied body after the claim succeeds. Preserve the exact target, options, and preconditions accepted during approval.

Database selection then follows workload and failure requirements. A single-host service using a local database can be suitable for a bounded operating mode. Multi-host availability, recovery-point requirements, writer concurrency, and administrative separation may require another implementation.

Changing the database does not remove the external-effect gap. Re-run the behavioral tests against the new transaction, failover, and recovery semantics.

Release the Next Capability, Not the Entire Architecture

The lab supports continuing integration work. Before granting real execution authority, demonstrate the boundaries it intentionally omits.

An authenticated approval must bind the correct action. Competing deployed workers must not create independent execution rights. Stale workers must lose usable access when ownership changes. Lost responses must enter a tested reconciliation path. Restored state must not revive authority without independent acceptance.

Also prove permitted work completes. A system that preserves every approval forever but cannot recover legitimate operations is not a sustainable service.

Assign ownership explicitly. Evaluation engineering maintains the test cases and retained results. Platform engineering operates the ledger and execution service. Identity and security teams govern access and suspension. The target-system owner defines acceptable observations and duplicate handling. The service owner accepts the operating scope and reconciliation burden.

Track contention, stranded claims, unresolved-action age, and recovery decisions separately from model quality. A slow database and a poor AI recommendation need different repairs.

Conclusion

An AI agent execution ledger turns approval consumption into a recorded state transition instead of an assumption shared by several workers.

The important result is not simply that one process wins a race. It is that the committed claim remains meaningful after a worker exits, that uncertainty does not automatically replenish authority, and that recovery exposes missing history rather than treating a restored database as current truth.

The local lab demonstrates selected parts of that behavior and a counterexample the database cannot solve alone. Target execution, independent observations, authenticated approval, and recovery admission still need their own enforced boundaries.

The control plane must not grade itself. A restart must not give it another allowance.

Stop one worker immediately after its dispatch intent commits. Before a replacement acts, what evidence establishes that the approval has not already produced its permitted effect?

External References

• SQLite: Transaction
• SQLite: PRAGMA Statements
• SQLite: Appropriate Uses For SQLite
• Python: sqlite3, DB-API 2.0 interface for SQLite databases
• OWASP: Transaction Authorization Cheat Sheet
• Amazon Builders’ Library: Making retries safe with idempotent APIs
• Kubernetes: Kubernetes API Concepts

The post Building an AI Agent Execution Ledger That Survives Restarts appeared first on Digital Thought Disruption.