How to Design the Trusted Agent Controller: Enterprise Agent Control Plane Series, Part 2

TL;DR

A trusted agent controller is the deterministic authority around an AI agent. It owns the run, not the model.

The controller should bind authenticated identity to every task, maintain authoritative workflow state, evaluate versioned policy, pause for scoped approvals, enforce budgets and concurrency, coordinate retries and cancellation, terminate unsafe loops, and preserve evidence that proves what happened.

The model can interpret goals, propose plans, select possible tools, and construct candidate arguments. It should not be authoritative for identity, permissions, workflow status, approval validity, policy version, resource budget, execution success, or stop conditions.

The most useful design rule is:

Put probabilistic reasoning inside a deterministic state machine, then make every consequential transition pass through trusted code.

Introduction

Part 1 of this series established the architectural boundary: MCP is the governed tool plane, not the complete agent controller. The agent proposes, the controller authorizes, MCP carries the governed interaction, and the runtime executes and verifies.

That distinction is easy to agree with at the diagram level. The harder question is what the controller must actually contain.

A production agent controller is not just an orchestration loop that calls a model until the model says it is finished. It is the system that decides whether a run exists, whose authority it carries, which state is authoritative, what the agent may see, which actions are allowed, when approval is required, how much resource the run may consume, how failures are handled, and when execution must stop.

This is where many agent platforms remain too model-centric. They store the conversation, expose a list of tools, add a retry loop, and call the result an agent runtime. That may be sufficient for a demonstration. It is not sufficient for a service that can change production systems, communicate externally, move money, alter customer records, or initiate other automated work.

The trusted controller is the part of the architecture that converts an agent from a capable suggestion engine into a bounded enterprise service.

The Controller Owns the Run

The controller should be authoritative for the complete lifecycle of an agent run.

That lifecycle begins before the first model call and continues after the final tool response. It includes request validation, identity binding, scope calculation, state creation, plan evaluation, policy decisions, approvals, execution, verification, evidence retention, operator intervention, and final disposition.

The model participates in the run. It does not own it.

A useful way to test the boundary is to ask what should still work correctly if the model is unavailable, manipulated, confused, or returning invalid output. The controller should still be able to:

• reject an unauthenticated request
• determine the correct tenant and environment
• withhold tools that are outside the approved scope
• refuse an action that violates policy
• preserve a paused workflow
• enforce an expired approval
• stop when the budget is exhausted
• suppress a duplicate side effect
• cancel pending work
• verify whether an external change actually occurred
• reconstruct the run for an operator

If any of those controls depend on the model choosing to comply, the authority boundary is in the wrong place.

The Trusted Controller Is a Logical Boundary

The phrase trusted agent controller can sound like one large application. It does not need to be.

In a small implementation, the controller may be one service with a database, policy library, queue, and MCP client. In a larger platform, it may be a logical control plane composed of several services:

• request and task API
• identity context service
• workflow state service
• policy decision point
• approval service
• budget and quota service
• scheduler and concurrency manager
• tool catalog and MCP gateway
• execution coordinator
• verification service
• evidence and audit store
• operator console and kill-switch service

The important requirement is not physical consolidation. It is authoritative coordination.

There must be one clear run identifier, one authoritative state model, one policy decision path, one definition of the active approval, one budget ledger, and one final status. Splitting components into services must not create several competing versions of the truth.

Reference Architecture for the Trusted Agent Controller

The controller sits between the business request and the agent, then between the agent proposal and execution. The model is called from inside the control boundary rather than placed in front of it.

The key pattern is the second controller check after the model responds. The model never moves directly from reasoning to execution. It returns a proposal, and the controller decides whether that proposal can cause a state transition.

Core Controller Modules

This decomposition keeps the model useful without allowing it to become the policy engine, database, scheduler, approval authority, or source of operational truth.

Identity Context Must Be Bound Before Planning

Identity is not a field the model should infer from the conversation.

The controller should establish the identity context before it constructs the model prompt or exposes any tool catalog. At minimum, distinguish three identities.

Requester identity

The human, service, event source, or upstream workflow that initiated the task.

Agent workload identity

The non-human identity used by the controller or execution runtime to authenticate to platform services.

Downstream execution identity

The identity or delegated authority presented to the target resource, API, MCP server, or enterprise system.

These identities may be related, but they should not be collapsed into one shared credential.

A trusted identity context should contain values such as:

• requester subject and organization
• agent or workflow identity
• tenant and data boundary
• granted scopes or roles
• approved environment
• delegation mode
• authentication strength
• token audience
• session and credential expiration
• source application
• trace and run identifiers

The controller should treat this context as immutable for the current authorization decision. If identity or scope changes, create a new decision and usually a new execution envelope.

The model may reference identity context in reasoning, but it should not be able to rewrite it. A tool argument such as is_admin: true or environment: production must never become authoritative because the model emitted it.

Durable Workflow State Must Be Separate from Model Context

A transcript is not a workflow database.

Model context is a temporary view assembled for a reasoning step. Authoritative state is the durable record used to decide what has happened, what is allowed next, and whether a side effect may execute.

The controller should separate at least these state classes:

A robust controller commonly uses both a current state record and an append-only event history. The current record makes decisions efficient. The event history makes the run reconstructable.

Every state update should include a version or sequence number. If two workers attempt to advance the same run, only one should succeed. This prevents duplicate execution caused by races, retries, delayed messages, or repeated queue delivery.

Use an Explicit State Machine

The state machine should define which transitions are legal. Do not let the model invent a new lifecycle state or skip a control gate.

The exact states will vary by workflow, but several principles should remain consistent:

• authorization occurs before execution
• approval is a durable waiting state, not a user-interface pause
• execution and verification are separate states
• cancellation is requested first and confirmed only after work is contained
• a failed action and a safely contained failure are different outcomes
• every terminal state has a reason code

MCP Tasks can help represent deferred protocol operations, but the controller’s business workflow state remains broader. One run may contain several MCP calls, human approvals, timers, model calls, and non-MCP work. The controller should map protocol tasks into its state model rather than replacing the complete state model with one protocol task.

Policy Decisions Must Be Deterministic and Versioned

The model can explain policy. It should not be the final policy decision point for consequential actions.

A controller policy decision should evaluate structured inputs such as:

• authenticated requester
• agent workload identity
• task type
• target resource
• target environment
• proposed operation
• data classification
• action consequence
• reversibility
• current workflow state
• approval status
• policy version
• remaining budget
• time window
• MCP server and tool trust status

The result should also be structured:

• allow
• deny
• require approval
• constrain arguments
• reduce capability
• route to a safer workflow
• escalate for manual handling

Every result should include a policy version, decision reason, evaluated facts, and obligations that must be enforced later.

For example, an allow decision may carry obligations such as:

• use only a production-read scope
• redact customer identifiers before a model call
• require approval before a write
• execute with zero automatic retries
• verify the result through a separate read capability
• retain evidence for a defined period

This is more useful than a simple Boolean. The controller can carry those obligations into the execution envelope and verification stage.

Retrieved content and tool results must not be able to redefine policy. They may provide evidence used by a policy decision, but they are not themselves policy authorities.

Approvals Must Authorize an Exact Action

A generic approval such as “allow the agent to continue” is too broad.

The approval record should bind to the exact decision being authorized. A practical approval object includes:

• run ID
• workflow and policy versions
• requester identity
• agent identity
• target resource and environment
• operation
• normalized arguments or an argument hash
• expected side effects
• evidence shown to the approver
• risk classification
• approval reason
• approver identity and authority
• decision time
• expiration time
• single-use status

If the proposed arguments, target, tool version, policy version, or environment changes materially after approval, the approval should become invalid.

The controller must also close alternate paths. An approval-required action must not be executable through another tool, a retry path, an agent handoff, or a resumed session without the same valid approval decision.

When a workflow resumes, revalidate:

• requester and agent identity
• granted scopes
• approval expiration
• target state
• policy version
• tool version
• current budget
• concurrency lease

Approval does not freeze the environment. It authorizes a bounded decision under stated conditions.

Budgets Must Cover the Complete Run

Agent cost and risk are workflow properties, not model-call properties.

The controller should maintain a budget ledger that covers more than tokens. Useful budget dimensions include:

• model calls
• input and output tokens
• reasoning effort when exposed
• tool calls
• retries
• retrieved documents or bytes
• tool-result size
• wall-clock time
• queue time
• approval wait time
• parallel branches
• sandbox CPU and memory
• external API usage
• financial spend
• number of side effects
• number of high-impact actions

A mature controller should enforce both resource budgets and authority budgets.

A resource budget limits consumption. An authority budget limits how much change the run may produce. For example, a workflow may be allowed to restart one service instance, update ten records, send one external message, or create one change request. Even if model and token budget remains available, the authority budget can stop further side effects.

Budget exhaustion should produce a defined transition such as pause, degrade, escalate, or stop. It should not silently reset because the run was retried, resumed, or moved to another worker.

Budget reservations should happen before expensive or consequential work. The controller can reserve an amount, execute the action, then reconcile actual usage against the reservation.

Retries Must Be Classified and Idempotent

A generic retry loop is one of the fastest ways to duplicate agent side effects.

The controller should classify failures before deciding whether to retry.

Every retryable side effect should use an idempotency key created by trusted code. The key should bind to the run, operation, target, and normalized intent.

The model should not create the key because the model cannot be trusted to determine whether two requests represent the same side effect.

The controller should also detect:

• repeated proposals
• repeated tool arguments
• no-state-change loops
• cyclic handoffs
• plans that consume resources without advancing the workflow
• retries that attempt to bypass a previous denial

Cancellation Is a Protocol, Not a Button

Cancellation is often presented as an immediate user-interface action. In distributed execution, it is a coordinated state transition.

When an operator or policy requests cancellation, the controller should:

• stop scheduling new work
• signal cancellable model, MCP, queue, or runtime operations
• revoke or expire execution leases
• block new side effects for the run
• determine whether an in-flight action completed
• verify the authoritative system state
• run compensation when required and permitted
• preserve partial results and evidence
• record the cancellation reason and actor
• transition to canceled only when containment is confirmed

MCP includes cancellation and progress mechanisms, and Tasks can expose status for deferred operations. Those protocol features are useful, but the controller still needs end-to-end cancellation semantics across the complete workflow.

A canceled client request does not prove that a downstream action stopped. A timeout does not prove that it failed. The controller must reconcile the final state.

Termination Rules Must Be Independent of the Model

The model should not be the only component allowed to declare that a run is finished.

The controller should terminate or pause a run when any of these conditions occur:

• terminal workflow state reached
• critical policy violation
• approval rejected or expired
• budget exhausted
• maximum elapsed time reached
• maximum model or tool calls reached
• no-state-change threshold reached
• repeated proposal threshold reached
• duplicate side-effect risk detected
• verification failed
• dependency circuit breaker opened
• operator kill switch activated
• identity or authorization revoked
• policy version became incompatible
• target environment changed outside approved assumptions

Every termination should use a structured stop reason. “The model stopped responding” is not enough for operations, reporting, or evaluation.

Kill switches should exist at several levels:

• agent release
• task type
• workflow instance
• MCP server
• individual tool
• target environment
• credential
• downstream route

Operators should not need to redeploy the complete platform to disable one unsafe capability.

Verification Must Decide Whether the Outcome Occurred

A successful tool response is not the same thing as a successful workflow.

The verification service should compare the observed state of the authoritative system with the expected post-condition. Depending on the workflow, verification may include:

• reading the target resource after a write
• checking a job or task status
• comparing record versions
• confirming a message was accepted by the intended system
• testing service health
• validating generated artifacts
• checking that only approved fields changed
• confirming that no duplicate action occurred
• measuring whether the requested business outcome was achieved

Verification should be independent where practical. If one tool performed the change, another read path or direct system query should confirm it.

The model can summarize evidence, but trusted code should decide whether required conditions passed.

The controller should distinguish:

• executed and verified
• executed but partially verified
• outcome uncertain
• execution failed before side effect
• side effect occurred but workflow failed
• compensated successfully
• compensation failed

Those states matter during incidents and recovery.

Evidence Must Reconstruct the Control Path

The controller should preserve enough evidence for an operator to answer:

• Who requested the action?
• Which agent and release handled it?
• What context was presented to the model?
• What proposal did the model return?
• Which policy version evaluated it?
• Why was it allowed, denied, constrained, or sent for approval?
• Who approved it, and what exactly did they approve?
• Which MCP server and tool version executed?
• Which credentials and scopes were used?
• What did the target system report?
• How was the result verified?
• What budget was consumed?
• Why did the run stop?

Use one trace or run identifier across controller services, model calls, policy decisions, approval events, MCP calls, downstream actions, and verification.

OpenTelemetry provides a useful common telemetry foundation, and current GenAI conventions cover model and tool operations. The controller will still need domain-specific events for policy, approval, state transition, budget, and verification decisions.

Do not make incident response depend on private model reasoning. Record structured proposals, decision summaries, selected evidence, state transitions, and policy outcomes instead.

Sensitive content capture should be disabled by default. Prompts, tool arguments, tool results, retrieved data, and artifacts can contain secrets, customer information, source code, and regulated records.

Example Controller Contract

The following YAML is a vendor-neutral application contract, not an MCP protocol object. It shows how a controller can keep proposed work separate from authoritative identity, policy, and execution controls.

controller_contract:
  version: 4

  workflow:
    type: production_service_remediation
    state_schema: remediation-run-v3
    allowed_terminal_states:
      - succeeded
      - denied
      - canceled
      - failed_safe
      - manual_handoff

  identity:
    requester_context_required: true
    workload_identity_required: true
    delegated_access_mode: on_behalf_of
    token_audience_validation: true
    model_may_override_identity: false

  policy:
    policy_version: agent-control-policy-v12
    default_decision: deny
    production_changes_require_approval: true
    high_impact_actions_require_separation_of_duties: true
    retrieved_content_may_change_policy: false

  approvals:
    bind_to_argument_hash: true
    bind_to_target: true
    bind_to_tool_version: true
    single_use: true
    revalidate_before_execution: true
    maximum_age_minutes: 30

  budgets:
    max_model_calls: 6
    max_tool_calls: 8
    max_retries_total: 2
    max_elapsed_seconds: 300
    max_parallel_branches: 2
    max_change_actions: 1
    max_external_messages: 0

  execution:
    require_idempotency_key: true
    require_precondition_check: true
    require_postcondition_check: true
    retry_unknown_outcome: false
    cancellation_requires_reconciliation: true

  observability:
    trace_complete_run: true
    record_state_transitions: true
    record_policy_decisions: true
    record_approval_events: true
    record_budget_events: true
    capture_sensitive_content: false

  termination:
    no_state_change_limit: 2
    repeated_proposal_limit: 2
    stop_on_policy_change: true
    stop_on_identity_revocation: true
    operator_kill_switch: true

The values are examples. Production limits should be based on the actual workflow consequence, service objectives, historical behavior, and evaluation evidence.

Successful execution means every run can prove which contract version applied. What can go wrong is equally important: an approval may become stale, a tool may not support idempotency, a budget may be too low for legitimate recovery, or a policy may produce too many manual escalations.

Treat the contract as versioned production configuration, not a static checklist.

Example Deterministic Controller Loop

The controller loop below is simplified Python-like code. It demonstrates the sequence, not a complete framework.

from dataclasses import dataclass
from enum import Enum
from typing import Any


class Decision(str, Enum):
    ALLOW = "allow"
    DENY = "deny"
    REQUIRE_APPROVAL = "require_approval"


@dataclass(frozen=True)
class Proposal:
    tool_name: str
    arguments: dict[str, Any]
    rationale: str
    expected_postcondition: dict[str, Any]


def advance_run(run_id: str) -> None:
    # Load under an optimistic concurrency token or database lock.
    run = state_store.load_for_update(run_id)

    if run.is_terminal:
        return

    identity = identity_service.resolve(run.request)
    policy_context = policy_service.build_context(run, identity)

    allowed_tools = tool_catalog.for_context(policy_context)
    bounded_context = context_builder.for_next_step(
        run,
        allowed_tools,
    )

    proposal: Proposal = model_adapter.propose(bounded_context)
    proposal_validator.validate(proposal, allowed_tools)

    decision = policy_service.evaluate(
        context=policy_context,
        proposal=proposal,
        policy_version=run.policy_version,
    )

    evidence_store.record_policy_decision(run.id, decision)

    if decision.result == Decision.DENY:
        state_store.transition(
            run,
            "denied",
            reason=decision.reason,
        )
        return

    if decision.result == Decision.REQUIRE_APPROVAL:
        approval = approval_service.create_scoped_request(
            run=run,
            proposal=proposal,
            obligations=decision.obligations,
        )
        state_store.transition(
            run,
            "waiting_approval",
            approval_id=approval.id,
        )
        return

    budget_service.reserve(
        run.id,
        proposal,
        decision.obligations,
    )

    envelope = execution_envelope.build(
        run=run,
        identity=identity,
        proposal=proposal,
        obligations=decision.obligations,
    )

    state_store.transition(run, "executing")
    result = executor.invoke_mcp(envelope)

    verification = verifier.check(
        expected=proposal.expected_postcondition,
        execution_result=result,
        authoritative_target=envelope.target,
    )

    evidence_store.record_execution(
        run.id,
        result,
        verification,
    )

    if verification.passed:
        state_store.transition(run, "succeeded")
    else:
        recovery_service.contain_or_compensate(
            run,
            envelope,
            result,
        )

The implementation must add timeouts, cancellation, lease handling, idempotency, approval resume behavior, error classification, credential management, telemetry, and secure storage.

The important point is the order:

• tool exposure is filtered before the model call
• the proposal is validated after the model call
• policy runs before execution
• budget is reserved before a side effect
• verification happens after execution
• state transitions are performed by trusted code

Common Controller Failure Modes

A Practical Implementation Sequence

Do not begin by implementing every possible autonomous workflow. Build the control plane around one narrow task class.

Define the task contract

Specify the accepted request, target resources, allowed outcomes, prohibited outcomes, data classification, risk tier, and business owner.

Define the state machine

List legal states, transitions, terminal outcomes, pause points, cancellation behavior, and resume rules before writing the agent loop.

Bind identity and environment

Resolve requester, agent, tenant, scopes, environment, and delegation before exposing tools or building model context.

Create a minimal policy model

Start with deterministic rules for allowed tools, environments, action classes, approval requirements, and budgets. Return structured reasons and obligations.

Add a scoped approval object

Bind approval to the exact action, target, arguments, evidence, policy version, and expiration.

Add budget and concurrency ledgers

Track model calls, tool calls, retries, elapsed time, parallelism, and side effects in durable state.

Integrate the MCP tool plane

Expose only approved servers and tools. Validate schemas, inject trusted execution context, use separate downstream credentials, and preserve trace correlation.

Add independent verification

Define post-conditions before enabling write authority. Prove the target system reached the expected state.

Add cancellation and operator controls

Test pause, resume, cancel, disable, quarantine, credential revocation, and tool kill switches.

Run failure-injection tests

Inject timeouts, duplicate messages, policy changes, approval expiration, stale state, tool errors, partial completion, identity revocation, and budget exhaustion.

Expand authority gradually

Begin with read, simulate, or propose modes. Enable bounded changes only after trace evidence shows the controller can enforce the intended limits.

Operating Metrics for the Controller

The controller should be operated as a production service. Measure signals that reveal whether control is working:

• accepted and rejected runs by task type
• policy allow, deny, constrain, and approval rates
• approval wait time and expiration rate
• model proposals rejected by schema or policy
• tool calls per successful task
• retries and unknown-outcome events
• duplicate actions suppressed
• no-state-change and loop terminations
• budget exhaustion by dimension
• cancellation request to containment latency
• verification failure rate
• compensation success and failure rate
• manual handoff rate
• cost per successful task
• state-resume failures
• stale-version resume blocks
• operator kill-switch activations

High denial rates may indicate malicious activity, but they may also indicate that the tool catalog, workflow contract, or policy is poorly designed.

High approval volume may look safe while creating reviewer fatigue. A low retry rate may indicate reliability, or it may mean failures are being hidden. Metrics need operational interpretation.

Ownership Model

Distributed ownership is necessary, but one owner must remain accountable for the complete service outcome.

Trusted Agent Controller Checklist

Identity

• Requester identity is authenticated and bound to the run.
• Agent workload identity is separate from the requester.
• Tenant, scopes, environment, and delegation are supplied by trusted code.
• Downstream tokens are audience-bound and short-lived.
• The model cannot create or override authority fields.

State

• The workflow uses a typed, versioned state machine.
• Model context is not the authoritative state store.
• State transitions are atomic and concurrency-safe.
• Pause, resume, and migration behavior are defined.
• An append-only history can reconstruct the run.

Policy and approval

• Policy decisions are deterministic, versioned, and explainable.
• Decisions carry obligations, not only allow or deny.
• Approvals bind to the exact action, target, arguments, and versions.
• Approval expiration and single-use behavior are enforced.
• Resumed work is revalidated before execution.

Budgets and execution

• One durable ledger tracks the complete run.
• Resource and authority budgets are both enforced.
• Retry rules are defined by failure class.
• Retryable side effects require trusted idempotency keys.
• Unknown outcomes are reconciled before retry.
• Concurrency, queueing, and lease behavior are controlled.

Cancellation and termination

• Cancellation stops new work and reconciles in-flight work.
• Terminal states have structured reason codes.
• Loop, duplicate, budget, policy, and elapsed-time stop rules exist.
• Agent, workflow, tool, environment, and credential kill switches are tested.

Verification and evidence

• Expected post-conditions are defined before execution.
• The authoritative system is queried after side effects.
• Policy, approval, tool, result, and verification evidence share one run ID.
• Sensitive content capture is restricted and governed.
• Operators can reconstruct a run without private model reasoning.

Conclusion

A trusted agent controller is the authority boundary that makes an enterprise agent governable.

It should own identity, durable state, policy decisions, approvals, budgets, concurrency, retries, cancellation, termination, verification, and evidence. The model remains valuable because it can interpret intent, reason across context, propose plans, and construct candidate actions. It becomes safer because trusted code decides which proposals can advance the workflow.

The controller does not need to be one monolithic service, but it does need one coherent control model. Every run must have one authoritative identity context, one state machine, one policy path, one budget ledger, one approval record, one trace, and one final disposition.

The practical design principle is straightforward:

Put probabilistic reasoning inside a deterministic state machine, and make every consequential transition pass through trusted code.

Part 3 of this series will move below the controller into the MCP gateway and server layer. It will cover server admission, tool filtering, schema governance, credential separation, routing, domain authorization, error contracts, and MCP-specific operational controls.

External References

• Model Context Protocol: Architecture overview
  Canonical URL: https://modelcontextprotocol.io/docs/learn/architecture
• Model Context Protocol: Specification
  Canonical URL: https://modelcontextprotocol.io/specification/2025-11-25
• Model Context Protocol: Tasks
  Canonical URL: https://modelcontextprotocol.io/specification/2025-11-25/basic/utilities/tasks
• Model Context Protocol: Cancellation
  Canonical URL: https://modelcontextprotocol.io/specification/2025-11-25/basic/utilities/cancellation
• Model Context Protocol: Authorization
  Canonical URL: https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization
• Model Context Protocol: Security Best Practices
  Canonical URL: https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices
• National Institute of Standards and Technology: AI Risk Management Framework
  Canonical URL: https://www.nist.gov/itl/ai-risk-management-framework
• National Institute of Standards and Technology: NIST AI RMF Playbook
  Canonical URL: https://airc.nist.gov/airmf-resources/playbook/
• OWASP Gen AI Security Project: OWASP Top 10 for Agentic Applications for 2026
  Canonical URL: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
• OpenTelemetry: Inside the LLM Call: GenAI Observability with OpenTelemetry
  Canonical URL: https://opentelemetry.io/blog/2026/genai-observability/

The post How to Design the Trusted Agent Controller: Enterprise Agent Control Plane Series, Part 2 appeared first on Digital Thought Disruption.