OpenTelemetry for AI Agents: Trace Models, Tool Calls, Tokens, Latency, and Cost

TL;DR

AI agent observability requires more than a request log and a model latency chart. A single agent invocation can contain planning, multiple model calls, tool execution, retries, retrieval, memory operations, and nested agents. OpenTelemetry’s developing Generative AI semantic conventions now provide a common vocabulary for many of those operations, including agent, workflow, model, and tool spans plus token and latency metrics.

The practical implementation pattern is to trace the complete agent invocation, record every model and client-side tool call as a child operation, use provider-returned billable token counts when available, and export the telemetry through OTLP. Prompt content, completions, tool arguments, and tool results should remain disabled by default because they can expose sensitive data.

Cost should be calculated from standardized usage telemetry and a versioned pricing table. It should not be presented as a native OpenTelemetry GenAI cost convention when it is actually an organization-specific estimate.

Introduction

An AI agent takes 24 seconds to answer a question. The application reports HTTP 200, the model provider reports no outage, and the final answer looks correct.

That still leaves the operations team with the questions that matter:

• Did the agent call the model once or six times?
• Did a tool wait on an external API?
• Did the orchestrator retry after a transient failure?
• Did the prompt consume 2,000 input tokens or 80,000?
• Did the model return a tool request, a completed answer, or a truncated response?
• Did the request cost a fraction of a cent or several dollars?
• Can the team compare that behavior across agents, models, prompt versions, and environments?

Traditional application monitoring usually sees the outer request and perhaps the provider API call. It does not automatically explain the agent’s internal decision path. Agent frameworks can add even more ambiguity by hiding loops, retries, and tool routing behind convenient abstractions.

OpenTelemetry is becoming the common telemetry layer for this problem. Its Generative AI semantic conventions define shared names for model operations, agent invocations, tool execution, token usage, latency, prompt metadata, response metadata, and related signals. The important caveat is that these conventions are still marked Development. Teams should adopt them deliberately, pin their instrumentation dependencies, and expect controlled schema evolution.

This article builds a practical observability model and a working Python lab. The goal is not simply to generate spans. The goal is to make agent behavior explainable enough for developers, architects, operators, security teams, and FinOps owners to act on the same evidence.

Why Traditional APM Stops Too Early

A conventional distributed trace often ends with a span such as POST /chat and a child HTTP request to a model endpoint. That is useful, but it treats the AI application as a black box.

An agent request is closer to a small distributed workflow:

1. The agent receives an objective.
2. The orchestrator selects a prompt and model.
3. The model decides whether it can answer or needs a tool.
4. The application executes the tool.
5. The tool result becomes new model context.
6. The model produces a final response.
7. The runtime may repeat parts of the sequence because of retries, validation failures, or planning logic.

The outer HTTP span can be healthy while the agent is operationally unhealthy. A retry loop may inflate latency and token usage. A tool may return a technically valid but semantically useless response. A prompt change may double input tokens without changing answer quality. A provider alias may silently route to a different response model.

The observable unit therefore cannot be only the web request or only the model call. It must include the complete agent invocation and the operations nested beneath it.

What OpenTelemetry Standardizes for AI Agents

The GenAI conventions provide a shared vocabulary, not a complete operating model. They tell producers and backends how to describe common operations so that telemetry from different frameworks and providers can be compared more consistently.

There are two consequences for enterprise architecture.

First, instrumentation should use the standard names when the conventions define the operation. Custom attributes are still useful, but they should extend the model rather than replace it.

Second, a standards-based trace is not automatically a safe trace. Some defined fields can contain prompts, user data, tool parameters, secrets, or returned business records. The existence of a semantic convention does not mean every field should be enabled in production.

The Trace Model That Makes Agent Behavior Visible

The most useful trace starts at the agent boundary and preserves parent-child relationships through every model and tool operation.

The diagram below shows a single request in which the first model call requests a tool, the application executes that tool, and a second model call produces the final answer.

The parent span answers the service-level question: how long did the agent take from invocation to its final response?

The model spans answer the inference questions: which provider and model were requested, which model actually responded, how many tokens were consumed, why did generation stop, and did an error occur?

The tool span answers the integration question: what capability was invoked, how long did it take, and was the failure inside the tool path rather than the model path?

This structure also exposes loops. When a trace contains nine model spans and eight tool spans for a simple request, the team can see the behavior instead of trying to infer it from aggregate token totals.

Model Retries Belong Inside the Logical Operation

The model span convention treats a model operation as a logical client operation. When the provider SDK automatically retries a transient failure, the span should cover the full logical operation, including those retries.

That keeps end-user latency accurate, but it creates an implementation decision. Teams that need individual retry visibility can add retry events or lower-level protocol spans without breaking the logical model span. The outer GenAI span should remain the stable business-facing operation.

Nested Agents Need Separate Invocation Boundaries

A parent agent that delegates to another agent should not flatten every downstream call into one unstructured trace. Each agent invocation should have its own span and its own inference-call and tool-call counts.

This allows the platform team to answer a more useful question: which agent owns the model or tool activity? It also prevents calls made by sub-agents from being counted twice against the parent invocation.

Define a Telemetry Contract Before Writing Instrumentation

OpenTelemetry gives teams attribute names, but the organization still needs a telemetry contract. Without one, every application emits slightly different values and the dashboards fragment.

A practical contract should define:

Do not substitute random values when an attribute is unavailable. A generated conversation identifier that exists only for telemetry creates false correlation. An invented provider name destroys cross-application consistency. An unbounded user identifier in metrics creates cardinality and privacy problems.

The contract should also identify which team owns each field. Application teams usually own the agent, prompt, and tool names. Platform teams own service and environment metadata. Provider integrations own requested and response model parsing. Security and governance teams own content-capture policy.

A Working Python Lab

This lab produces a trace containing one agent span, two model spans, and one tool span. It also emits the standard model, token, agent, and tool duration histograms.

The model and tool operations are simulated. That keeps the lab runnable without provider credentials while preserving the instrumentation pattern that a real application should use.

What You Will Build

By the end of the lab, you will be able to:

• Export traces and metrics over OTLP.
• See an invoke_agent parent span with model and tool children.
• Record input and output token usage.
• Compare model-call, tool-call, and end-to-end agent duration.
• Validate the exact telemetry before adding a real provider SDK.
• Keep tool arguments and results disabled unless content capture is explicitly enabled.

Prerequisites

You need:

• Python 3.10 or later
• Docker
• A local shell
• Permission to expose local ports 18888, 4317, and 4318
• A disposable development environment

The dashboard configuration below allows anonymous local access. Do not reuse that security setting for a shared or production deployment.

Start the Local Telemetry Dashboard

Run the Aspire Dashboard container:

docker run --rm 
  -p 18888:18888 
  -p 4317:18889 
  -p 4318:18890 
  -d 
  --name aspire-dashboard 
  -e ASPIRE_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true 
  mcr.microsoft.com/dotnet/aspire-dashboard:latest

This exposes an OTLP gRPC receiver on local port 4317, an OTLP HTTP receiver on local port 4318, and the dashboard on local port 18888.

Install the Python Packages

Create a virtual environment and install the OpenTelemetry API, SDK, and gRPC OTLP exporter:

python -m venv .venv

# Linux or macOS
source .venv/bin/activate

# Windows PowerShell
.venvScriptsActivate.ps1

python -m pip install --upgrade pip
python -m pip install 
  opentelemetry-api 
  opentelemetry-sdk 
  opentelemetry-exporter-otlp-proto-grpc

Create the Instrumented Agent

Save the following as agent_observability_lab.py.

The script configures one resource, one trace pipeline, and one metric pipeline. It records important low-cardinality attributes when spans are created so a sampler can use them. It records token counts after the simulated provider response is available.

from __future__ import annotations

import json
import os
import random
import time
import uuid
from dataclasses import dataclass

from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
    OTLPMetricExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import SpanKind, Status, StatusCode


OTLP_ENDPOINT = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "localhost:4317")
CAPTURE_CONTENT = (
    os.getenv("OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "false")
    .strip()
    .lower()
    == "true"
)

AGENT_NAME = "incident-triage"
PROVIDER_NAME = "example.gen_ai"
REQUEST_MODEL = "example-chat-model"
RESPONSE_MODEL = "example-chat-model-2026-07"


resource = Resource.create(
    {
        "service.name": "dtd-ai-agent-lab",
        "service.version": "1.0.0",
        "deployment.environment.name": "lab",
    }
)

trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint=OTLP_ENDPOINT, insecure=True)
    )
)
trace.set_tracer_provider(trace_provider)

metric_reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint=OTLP_ENDPOINT, insecure=True),
    export_interval_millis=5_000,
)
meter_provider = MeterProvider(
    resource=resource,
    metric_readers=[metric_reader],
)
metrics.set_meter_provider(meter_provider)

tracer = trace.get_tracer("dtd.ai.agent")
meter = metrics.get_meter("dtd.ai.agent")

model_duration = meter.create_histogram(
    "gen_ai.client.operation.duration",
    unit="s",
    description="GenAI operation duration.",
)
token_usage = meter.create_histogram(
    "gen_ai.client.token.usage",
    unit="{token}",
    description="Number of input and output tokens used.",
)
agent_duration = meter.create_histogram(
    "gen_ai.invoke_agent.duration",
    unit="s",
    description="End-to-end duration of one agent invocation.",
)
agent_inference_calls = meter.create_histogram(
    "gen_ai.invoke_agent.inference_calls",
    unit="{inference_call}",
    description="Inference calls made during one agent invocation.",
)
agent_tool_calls = meter.create_histogram(
    "gen_ai.invoke_agent.tool_calls",
    unit="{tool_call}",
    description="Client-side tool calls made during one agent invocation.",
)
tool_duration = meter.create_histogram(
    "gen_ai.execute_tool.duration",
    unit="s",
    description="Duration of one tool execution.",
)


@dataclass(frozen=True)
class ModelResult:
    input_tokens: int
    output_tokens: int
    finish_reason: str


def record_error(span: trace.Span, exc: Exception) -> None:
    error_type = type(exc).__name__
    span.record_exception(exc)
    span.set_attribute("error.type", error_type)
    span.set_status(Status(StatusCode.ERROR, str(exc)))


def call_model(
    input_tokens: int,
    output_tokens: int,
    finish_reason: str,
) -> ModelResult:
    span_attributes = {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": PROVIDER_NAME,
        "gen_ai.request.model": REQUEST_MODEL,
    }

    started = time.perf_counter()
    error_type: str | None = None

    try:
        with tracer.start_as_current_span(
            f"chat {REQUEST_MODEL}",
            kind=SpanKind.CLIENT,
            attributes=span_attributes,
        ) as span:
            try:
                time.sleep(random.uniform(0.35, 0.85))

                span.set_attribute("gen_ai.response.model", RESPONSE_MODEL)
                span.set_attribute("gen_ai.usage.input_tokens", input_tokens)
                span.set_attribute("gen_ai.usage.output_tokens", output_tokens)
                span.set_attribute(
                    "gen_ai.response.finish_reasons",
                    [finish_reason],
                )

                return ModelResult(
                    input_tokens=input_tokens,
                    output_tokens=output_tokens,
                    finish_reason=finish_reason,
                )
            except Exception as exc:
                error_type = type(exc).__name__
                record_error(span, exc)
                raise
    finally:
        duration_attributes = {
            "gen_ai.operation.name": "chat",
            "gen_ai.provider.name": PROVIDER_NAME,
            "gen_ai.request.model": REQUEST_MODEL,
            "gen_ai.response.model": RESPONSE_MODEL,
        }
        if error_type:
            duration_attributes["error.type"] = error_type

        model_duration.record(
            time.perf_counter() - started,
            duration_attributes,
        )

        if error_type is None:
            common_token_attributes = {
                "gen_ai.operation.name": "chat",
                "gen_ai.provider.name": PROVIDER_NAME,
                "gen_ai.request.model": REQUEST_MODEL,
                "gen_ai.response.model": RESPONSE_MODEL,
            }
            token_usage.record(
                input_tokens,
                {
                    **common_token_attributes,
                    "gen_ai.token.type": "input",
                },
            )
            token_usage.record(
                output_tokens,
                {
                    **common_token_attributes,
                    "gen_ai.token.type": "output",
                },
            )


def search_cmdb(call_id: str) -> dict[str, str]:
    tool_attributes = {
        "gen_ai.operation.name": "execute_tool",
        "gen_ai.agent.name": AGENT_NAME,
        "gen_ai.tool.name": "search_cmdb",
        "gen_ai.tool.type": "function",
        "gen_ai.tool.call.id": call_id,
    }

    started = time.perf_counter()
    error_type: str | None = None

    try:
        with tracer.start_as_current_span(
            "execute_tool search_cmdb",
            kind=SpanKind.INTERNAL,
            attributes=tool_attributes,
        ) as span:
            try:
                arguments = {
                    "service": "payments-api",
                    "environment": "production",
                }

                if CAPTURE_CONTENT:
                    span.set_attribute(
                        "gen_ai.tool.call.arguments",
                        json.dumps(arguments),
                    )

                time.sleep(random.uniform(0.20, 0.55))

                result = {
                    "owner": "payments-platform",
                    "tier": "critical",
                    "runbook": "RB-1042",
                }

                if CAPTURE_CONTENT:
                    span.set_attribute(
                        "gen_ai.tool.call.result",
                        json.dumps(result),
                    )

                return result
            except Exception as exc:
                error_type = type(exc).__name__
                record_error(span, exc)
                raise
    finally:
        metric_attributes = {
            "gen_ai.agent.name": AGENT_NAME,
            "gen_ai.tool.name": "search_cmdb",
            "gen_ai.tool.type": "function",
        }
        if error_type:
            metric_attributes["error.type"] = error_type

        tool_duration.record(
            time.perf_counter() - started,
            metric_attributes,
        )


def invoke_agent() -> None:
    agent_attributes = {
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.agent.name": AGENT_NAME,
        "gen_ai.agent.version": "1.0.0",
    }

    started = time.perf_counter()
    inference_count = 0
    tool_count = 0
    error_type: str | None = None

    try:
        with tracer.start_as_current_span(
            f"invoke_agent {AGENT_NAME}",
            kind=SpanKind.INTERNAL,
            attributes=agent_attributes,
        ) as span:
            try:
                inference_count += 1
                call_model(
                    input_tokens=random.randint(1_100, 1_400),
                    output_tokens=random.randint(70, 120),
                    finish_reason="tool_calls",
                )

                tool_count += 1
                search_cmdb(call_id=f"call-{uuid.uuid4().hex[:8]}")

                inference_count += 1
                call_model(
                    input_tokens=random.randint(1_400, 1_700),
                    output_tokens=random.randint(180, 260),
                    finish_reason="stop",
                )
            except Exception as exc:
                error_type = type(exc).__name__
                record_error(span, exc)
                raise
    finally:
        invocation_attributes = {
            "gen_ai.agent.name": AGENT_NAME,
        }
        if error_type:
            invocation_attributes["error.type"] = error_type

        agent_duration.record(
            time.perf_counter() - started,
            invocation_attributes,
        )
        agent_inference_calls.record(
            inference_count,
            {"gen_ai.agent.name": AGENT_NAME},
        )
        agent_tool_calls.record(
            tool_count,
            {"gen_ai.agent.name": AGENT_NAME},
        )


def main() -> None:
    for _ in range(10):
        invoke_agent()

    trace_provider.shutdown()
    meter_provider.shutdown()


if __name__ == "__main__":
    main()

What You Must Change for a Real Agent

Replace the simulated call_model function with the provider or gateway SDK call. The instrumentation wrapper should extract:

• The exact requested model.
• The response model returned by the provider, when available.
• Provider-reported input and output token counts.
• Cache and reasoning token details when the provider exposes them.
• Finish reasons.
• Provider error codes or low-cardinality exception types.
• Request and response identifiers when they are operationally useful and permitted.

Use billable token counts when the provider exposes both consumed and billed units. Do not estimate tokens offline when authoritative usage is available from the provider response.

Replace search_cmdb with the actual tool boundary. Instrument the application-side execution, not merely the model’s decision to request the tool. The tool span should include a stable tool name and type. Arguments and results should remain opt-in because they can contain credentials, customer data, query text, ticket details, file paths, or system configuration.

Run the Lab

Execute the script:

python agent_observability_lab.py

Open the dashboard on local port 18888. Select the dtd-ai-agent-lab service.

A successful run should produce:

• Ten agent traces.
• One invoke_agent incident-triage root span per invocation.
• Two chat example-chat-model child spans per trace.
• One execute_tool search_cmdb child span per trace.
• Input and output token records.
• Model, agent, and tool duration histograms.
• Inference-call distributions centered on two.
• Tool-call distributions centered on one.

The exact trace durations and token values will vary because the lab intentionally introduces small random ranges.

Validate the Signal Before Building Dashboards

A dashboard can make incorrect telemetry look convincing. Validate the raw signal first.

Trace Validation Checklist

For one request, confirm:

• The agent span starts before its first child and ends after its last child.
• The two model calls and one tool call share the same trace.
• Each child has the agent span as its parent.
• The first model call ends with tool_calls.
• The final model call ends with stop.
• Requested and response model values are distinct fields.
• Token counts are integers and appear only after the simulated response.
• The tool duration matches the tool span duration closely.
• No prompt, tool argument, or tool result content appears unless capture was explicitly enabled.

Metric Validation Checklist

Confirm that:

• gen_ai.client.operation.duration is recorded once per model operation.
• gen_ai.client.token.usage is recorded separately for input and output token types.
• gen_ai.invoke_agent.duration represents the complete agent invocation.
• gen_ai.invoke_agent.inference_calls counts the agent’s model calls once.
• gen_ai.invoke_agent.tool_calls counts only client-side tool executions.
• gen_ai.execute_tool.duration is recorded once per observed tool call.
• Metric dimensions use bounded values such as known model, agent, provider, tool, and error names.

Do not add trace identifiers, prompt text, customer identifiers, tool arguments, or full exception messages as metric dimensions.

Build Cost from Usage, Not Guesswork

OpenTelemetry standardizes the usage signals needed for cost analysis, but a token count is not a price.

The practical cost pipeline joins telemetry with a governed rate card:

Estimated request cost =
    (billable input tokens / 1,000,000 * input token rate)
  + (billable output tokens / 1,000,000 * output token rate)
  + provider-specific cache or reasoning charges
  + external tool and infrastructure charges

The rate card should include at least:

Keep Estimated Cost Outside the Standard Namespace

Do not create an attribute such as gen_ai.cost and imply it is part of the standard convention unless a future specification explicitly defines it.

A better pattern is one of the following:

• Join token telemetry with pricing at query time.
• Enrich telemetry in a controlled processing pipeline.
• Write calculated cost into an organization-owned namespace.
• Store request-level cost in a FinOps dataset linked by trace and request identifiers.

An organization-owned field might use a name such as dtd.finops.estimated_cost, but the exact namespace should follow your enterprise telemetry governance standard.

Cost Is an Estimate Until Reconciled

Even accurate token telemetry may not match an invoice exactly. Differences can come from:

• Cached-token discounts.
• Batch or priority processing tiers.
• Reasoning-token billing.
• Minimum charges.
• Regional pricing.
• Enterprise commitments.
• Gateway markups.
• Provider-side tools.
• Currency conversion.
• Delayed billing corrections.

Use telemetry for allocation, anomaly detection, engineering decisions, and near-real-time guardrails. Reconcile against provider billing data for financial reporting.

Design Dashboards Around Operational Questions

A good dashboard does not begin with every available metric. It begins with the decisions operators need to make.

The local Aspire Dashboard is useful for confirming that telemetry is emitted correctly and for inspecting traces. A production platform should add retained dashboards, alerts, service-level objectives, access controls, and cost enrichment.

Use a Collector as the Production Control Point

Direct application-to-backend export is acceptable for a local lab. Production environments usually benefit from an OpenTelemetry Collector between workloads and observability backends.

The Collector becomes a telemetry policy enforcement point. It can normalize attributes, remove prohibited fields, route production and development data differently, and provide resilient export behavior.

It should not become the only place where application meaning is created. The application still knows the agent name, prompt release, tool boundary, requested model, and business operation. Emit that context at the source, then use the Collector to enforce cross-platform policy.

Control Content, Cardinality, and Sampling

Agent observability can become expensive or dangerous when teams capture everything without a contract.

Keep Content Capture Off by Default

Prompt and completion content can contain:

• Personal information.
• Credentials and tokens.
• Confidential documents.
• Source code.
• Security findings.
• Customer records.
• Tool arguments and returned system data.
• Hidden system instructions.

Metadata-only telemetry still provides substantial value. Model names, token counts, durations, tool names, finish reasons, prompt versions, and error types are enough for many reliability and cost questions.

When content is required for debugging or evaluation, apply explicit controls:

• Restrict capture to approved environments or sampled sessions.
• Redact secrets and regulated data before export.
• Separate content storage from standard operational telemetry when appropriate.
• Apply shorter retention.
• Encrypt data in transit and at rest.
• Limit access to authorized engineering and governance roles.
• Record who enabled capture and why.
• Test redaction against tool arguments and model output, not only user prompts.

Keep Metric Dimensions Low Cardinality

Good metric dimensions include:

• Agent name.
• Tool name.
• Tool type.
• Operation name.
• Provider name.
• Requested model.
• Response model.
• Environment.
• Low-cardinality error type.

Poor metric dimensions include:

• Trace ID.
• Conversation ID.
• User ID.
• Request ID.
• Full exception message.
• Prompt text.
• Tool argument values.
• Document identifier.
• Generated answer.

High-cardinality context belongs on spans or controlled logs, not in metric label sets.

Sample for Diagnostic Value

Head sampling is efficient but must decide before the trace completes. Important attributes such as agent name, operation name, provider, requested model, and tool type should therefore be present when the span is created.

Tail sampling can retain traces based on completed behavior, such as:

• Errors.
• Slow agent invocations.
• Excessive model calls.
• High token usage.
• High estimated cost.
• Policy violations.
• Rare tool paths.

Cost-aware tail sampling generally requires a processing layer that can evaluate completed traces or correlated usage data. Plan that capability rather than assuming a basic sampler can calculate price.

Common Failure Modes and Troubleshooting

One subtle problem is successful but incomplete instrumentation. A trace can look structurally correct while missing retries, server-side provider tools, asynchronous work, or calls made by a sub-agent. Compare the trace against the actual execution model and define known visibility boundaries.

A Practical Enterprise Adoption Path

Trying to capture every GenAI field on day one usually creates inconsistency and governance debt. A phased rollout is safer.

Establish the Metadata-Only Baseline

Start with:

• Service and environment resource attributes.
• Agent and workflow spans.
• Model spans.
• Tool spans.
• Requested and response models.
• Token counts.
• Finish reasons.
• Duration metrics.
• Low-cardinality errors.

Success means a developer can explain one slow request from the trace without reading application logs.

Add Release and Ownership Context

Add:

• Agent version.
• Prompt name and version.
• Tool owner.
• Application owner.
• Deployment release.
• Governed environment names.

Success means an operator can correlate a regression with a specific application, agent, prompt, or release.

Build Reliability Dashboards and Alerts

Create:

• Agent latency SLOs.
• Model latency comparisons.
• Tool latency and error alerts.
• Loop indicators based on model and tool calls per invocation.
• Token anomaly alerts.
• Slow and failed trace retention.

Success means the operations team can detect and isolate a regression before relying on user complaints.

Add Cost Enrichment and Budgets

Join usage telemetry with:

• Versioned provider rate cards.
• Contract and region metadata.
• Application and business ownership.
• Success or outcome signals.

Success means teams can compare cost per successful outcome, not merely total token spend.

Add Controlled Content and Evaluation Signals

Only after governance is established, add selected content or evaluation data for approved scenarios.

Success means the organization can debug quality problems without turning the observability platform into an uncontrolled copy of prompts, documents, and tool results.

Operational Ownership Matters More Than the Dashboard

Agent observability crosses several teams:

Without this ownership model, the dashboard becomes an attractive but unsupported artifact. The implementation is complete only when teams know who changes instrumentation, who approves content capture, who maintains pricing, and who responds when an alert fires.

Conclusion

OpenTelemetry gives AI agent teams a practical path out of framework-specific telemetry silos. A well-formed trace can show the complete agent invocation, every model call, every client-side tool execution, the tokens consumed, the latency introduced, and the errors encountered.

The standard is still developing, so adoption should be controlled rather than passive. Pin instrumentation versions, define a telemetry contract, validate raw signals, and preserve the distinction between standard GenAI attributes and organization-specific enrichment.

The most important implementation choice is the trace boundary. Start with the agent invocation, then make model, tool, retrieval, memory, and delegated-agent work visible beneath it. Keep sensitive content disabled by default. Use provider-returned billable usage when available. Derive cost from a governed, versioned rate card and reconcile it with billing data.

That combination turns observability from a debugging feature into an operating model. Developers can diagnose execution, architects can evaluate framework boundaries, operators can protect reliability, security teams can govern data capture, and FinOps teams can measure cost against actual agent behavior.

External References

• OpenTelemetry: open-telemetry/semantic-conventions-genai
  Canonical URL: https://github.com/open-telemetry/semantic-conventions-genai
• OpenTelemetry: Semantic Conventions for GenAI Agent and Framework Spans
  Canonical URL: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md
• OpenTelemetry: Semantic Conventions for Generative Client AI Spans
  Canonical URL: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md
• OpenTelemetry: Semantic Conventions for Generative AI Metrics
  Canonical URL: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md
• OpenTelemetry: Inside the LLM Call: GenAI Observability with OpenTelemetry
  Canonical URL: https://opentelemetry.io/blog/2026/genai-observability/
• OpenTelemetry: Python Instrumentation
  Canonical URL: https://opentelemetry.io/docs/languages/python/instrumentation/
• OpenTelemetry: Python Exporters
  Canonical URL: https://opentelemetry.io/docs/languages/python/exporters/

The post OpenTelemetry for AI Agents: Trace Models, Tool Calls, Tokens, Latency, and Cost appeared first on Digital Thought Disruption.