Introduction
A shared GPU platform creates a deceptively simple question: who used the GPU?
The question becomes difficult as soon as the platform supports more than one operating model. A Kubernetes pod may receive an entire GPU, a MIG instance, or a time-sliced share. A virtual machine may receive a vGPU profile or a pass-through device. A model server may serve requests for several customers from the same process. A scheduler may let one department borrow another department’s unused quota. The GPU may look busy while the application produces little useful work, or it may appear lightly utilized while holding scarce memory capacity that prevents another model from starting.
This is why a single GPU utilization graph cannot support tenant accountability. NVIDIA DCGM and DCGM Exporter are foundational components, but device telemetry does not automatically become department, project, model, or customer attribution [1]-[4]. Kubernetes metadata tells you which pod received a device, but not whether the pod produced valuable work [5], [6]. Model-server metrics expose tokens, requests, latency, and queue depth, but they do not establish who funded the underlying capacity [12]-[15]. A finance ledger can assign cost, but only after the platform has reliable allocation and identity evidence.
An enterprise AI-as-a-Service platform therefore needs an evidence chain, not a dashboard shortcut. That evidence chain must connect business ownership to scheduler decisions, workload identity, device identifiers, hardware telemetry, model-server outcomes, rate cards, and retained monthly records. It must also preserve uncertainty when attribution is approximate, especially for time-sliced GPUs, shared model processes, short-lived workloads, and unsupported hardware paths. Community discussions around emerging GPU observability tools reinforce the operational demand for attribution that goes beyond fleet-wide graphs [24].
TL;DR
A defensible GPU showback system measures four different things and never treats them as synonyms:
Allocation identifies who controlled or reserved GPU capacity during a time window.
Utilization measures what the GPU hardware was doing during sampled intervals.
Productive work measures the service outcome, such as successful requests, generated tokens, completed training steps, or processed images.
Financial consumption applies an approved cost policy to allocation, variable usage, shared overhead, and service outcomes.
Use DCGM and DCGM Exporter for device health and activity, Kubernetes PodResources or hypervisor inventory for ownership mapping, model-server metrics for workload outcomes, Prometheus for collection and recording rules, Grafana for role-specific dashboards, Alertmanager for actionable routing, and a separate financial evidence store for monthly showback or chargeback. Bill exact capacity from allocation records. Use utilization to expose efficiency. Use productive-work metrics to explain value. Mark process-level and time-slicing attribution with an explicit confidence level rather than presenting modeled estimates as exact measurements.
The Four Measurements That Must Not Be Collapsed
The most important design decision is conceptual, not technical. The platform must maintain separate definitions for allocation, utilization, productive work, and financial consumption.
Measurement
Primary question
Typical source
Useful unit
What it does not prove
Allocation
Who controlled capacity?
Scheduler, kubelet, device plugin, DRA, hypervisor, quota manager
GPU-hours, MIG-profile-hours, vGPU-profile-hours
That the device was busy
Utilization
What was the device doing?
DCGM, NVML, vGPU host metrics
SM active ratio, GPU busy ratio, memory used, watts
That useful business work completed
Productive work
What service outcome was delivered?
Triton, vLLM, NIM, training framework, application telemetry
Requests, tokens, samples, steps, jobs, successful outcomes
Who should pay for reserved capacity
Financial consumption
Which cost policy applies?
Rate card, asset model, scheduler ledger, finance rules
Currency, cost per GPU-hour, cost per request, cost per token
The physical reason for poor efficiency
Allocation
Allocation is a control-plane fact. It answers who had the right to use capacity, not whether they exercised that right efficiently.
For an exclusive Kubernetes allocation, the unit may be one GPU assigned to a pod for three hours. For MIG, the unit should preserve the profile, such as a specific MIG profile held for three hours. For vGPU, the unit should preserve the vGPU profile and VM assignment. For time-slicing, the unit should state the configured replica or share policy rather than pretending the tenant owned an isolated fraction of the silicon.
A general allocation equation is:
Allocated GPU-equivalent hours
= integral of assigned capacity fraction over elapsed time
The capacity fraction must come from an approved inventory model. It should not be inferred from a momentary utilization percentage.
Utilization
Utilization is sampled device behavior. Depending on the metric, it may describe the fraction of time the GPU was active, the fraction of streaming multiprocessors with at least one active warp, memory-copy engine activity, framebuffer memory consumption, power draw, or another device condition.
Utilization is useful for identifying idle reservations, saturation, poor batching, thermal throttling, memory pressure, or workload inefficiency. It is not a universal billing unit. A tenant may be financially responsible for a reserved GPU even when utilization is low, just as a reserved virtual machine still has a cost when its CPU is quiet.
An efficiency-oriented derived measure can be useful:
Active-equivalent GPU-hours
= integral of allocated capacity fraction
x normalized active ratio
over elapsed time
This is an analytical measure, not automatically an invoice measure. The organization must decide whether active-equivalent hours affect pricing, efficiency targets, or only operational reporting.
Productive Work
Productive work is application-specific. For an inference service, productive work may include successful requests, input tokens, output tokens, images generated, embeddings produced, or requests meeting a service-level objective. For training, it may include completed steps, samples processed, checkpoints produced, or jobs completed successfully. For interactive development, a productive outcome may be harder to automate and may require a different service model.
The same GPU activity can produce very different amounts of useful work depending on model size, precision, batch size, sequence length, runtime, cache behavior, concurrency, and latency target. This is why tokens per second, requests per second, time to first token, queue delay, and success rate must sit beside GPU telemetry.
Financial Consumption
Financial consumption applies policy to evidence. It determines whether a tenant pays for guaranteed quota, actual allocation, borrowed capacity, energy, premium support, model-serving consumption, or shared platform overhead.
A defensible chargeback model is explicit:
Tenant charge
= guaranteed capacity charge
+ borrowed allocation charge
+ variable service or energy charge
+ allocated shared-platform overhead
– approved credits and adjustments
The platform team should be able to explain every term. A finance reviewer should be able to reproduce the monthly result. An application owner should be able to see why an idle reservation still cost money, why borrowed usage was treated differently, and why high device utilization did not necessarily translate into a low cost per successful outcome.
Build an Attribution Chain, Not a Label Collection
Per-tenant telemetry depends on a stable chain of identities across business, orchestration, workload, device, and service layers.
Pod names, process identifiers, and local GPU indices are not durable enough by themselves. Pods are recreated. Process identifiers are reused. GPU index zero means something different on every node. Namespace names can be reused after deletion. Model names may remain unchanged while the weight digest, quantization, runtime profile, or deployment configuration changes.
The evidence model should use stable identifiers and preserve time validity.
Layer
Recommended evidence keys
Why they matter
Business
tenant ID, department ID, customer ID, cost center, project ID, service owner
Establishes accountability and reporting hierarchy
Kubernetes
cluster ID, namespace UID, workload controller UID, pod UID, container ID, node name
Survives pod-name reuse and connects device allocation to workload ownership
Physical GPU
GPU UUID, node, PCI bus identity, GPU model, serial where available
Establishes the physical device and prevents local-index ambiguity
MIG
parent GPU UUID, GPU instance ID, compute instance ID, MIG device UUID, profile
Identifies the allocated partition and its capacity class
Virtualization
hypervisor cluster, host ID, VM UUID, vGPU UUID, vGPU profile, guest instance ID
Connects host-side vGPU evidence to the correct VM and tenant
Model service
endpoint ID, model name, model version or digest, runtime, deployment revision
Connects hardware activity to the service delivering work
Request or job
trace ID, request ID, job ID, queue ID, user or service identity
Supports detailed investigation outside high-cardinality metric labels
Finance
rate-card version, allocation class, billing period, adjustment ID
Makes financial output reproducible
Preserve Time-Valid Ownership
The platform should maintain an ownership mapping with valid_from and valid_to timestamps. When a pod is deleted, a VM is reassigned, a namespace changes owners, or a vGPU moves, the old relationship must remain queryable for the period in which it was true.
This temporal mapping is essential for month-end close. A current CMDB lookup cannot reliably explain who owned a workload three weeks earlier. The evidence pipeline should capture assignment events as they happen and retain them independently from the live orchestration system.
Keep High-Cardinality Identity Out of Prometheus Labels
Prometheus labels are effective for bounded dimensions such as cluster, namespace, model, service tier, GPU class, and a manageable set of tenant identifiers. They are a poor place for request IDs, user email addresses, raw process IDs, session IDs, or arbitrary customer strings because every unique label combination creates another time series [8].
Use metrics for bounded aggregation and alerting. Use logs, traces, or an analytical event store for per-request and per-process detail. Join those records into the financial ledger during controlled aggregation rather than multiplying the number of Prometheus time series without limit.
Reference Telemetry Architecture
The reference architecture separates collection, identity enrichment, operational analytics, and financial evidence. Notice that Kubernetes and virtual-machine paths converge only after each has produced a reliable device-to-workload mapping.
What Each Component Owns
DCGM and DCGM Exporter own hardware telemetry and device health evidence.
Kubernetes PodResources, device allocation data, and hypervisor inventory own device-to-workload assignment evidence.
Kubernetes metadata and business mappings own workload-to-tenant attribution.
Model servers and applications own productive-work and service-level evidence.
Prometheus owns time-series collection, short-term query, recording rules, and alert-rule evaluation.
Grafana owns reusable operational and showback views, not the source-of-truth billing calculation.
Alertmanager owns deduplication, grouping, silencing, inhibition, and receiver routing for Prometheus alerts.
The financial ledger owns closed-period cost evidence, rate-card versions, adjustments, and reconciliation.
What DCGM and DCGM Exporter Actually Measure
NVIDIA Data Center GPU Manager provides the device-level telemetry, health, diagnostics, policy, profiling, and accounting capabilities that make GPU operations observable [2], [3]. DCGM Exporter exposes selected DCGM fields in Prometheus format and is commonly deployed as a DaemonSet on GPU nodes [1].
The exporter is a collection layer. It does not replace the identity model, scheduler ledger, or model-server telemetry.
Core Metric Domains
Domain
Example evidence
Operational use
Identity and inventory
GPU UUID, device model, node, MIG identifiers and profile
Device mapping, inventory reconciliation, capacity segmentation
Memory
framebuffer total, used, free, reserved, memory-copy activity
Model fit, memory pressure, stranded memory, leak detection
Compute activity
GPU active ratio, SM activity, SM occupancy or related profiling fields
Idle detection, saturation analysis, kernel efficiency clues
Power and thermals
power draw, temperature, memory temperature, clock values, clock-event reasons
Cooling issues, throttling, energy estimates, performance anomalies
Interconnect
PCIe and NVLink counters or throughput where supported
Data-path bottlenecks, topology issues, multi-GPU diagnosis
Reliability
XID incidents, ECC errors, row-remap or memory-repair indicators, device health
Incident response, quarantine, warranty and support evidence
Process and job accounting
process activity, process memory, job statistics where enabled and supported
Attribution refinement and workload summaries
Metric Names Are Versioned Interfaces
DCGM field names and exporter aliases evolve. A release may rename labels, add cumulative counters, change field aliases, or improve MIG and Kubernetes metadata behavior [4]. The platform should not let raw vendor metric names spread through every dashboard and financial query.
Use a normalization layer based on Prometheus recording rules or a telemetry transform. For example:
groups:
– name: ai-gpu-normalization
interval: 30s
rules:
– record: ai_gpu:sm_active_ratio
expr: clamp_max(DCGM_FI_PROF_SM_ACTIVE, 1)
– record: ai_gpu:framebuffer_used_bytes
expr: DCGM_FI_DEV_FB_USED * 1024 * 1024
– record: ai_gpu:power_watts
expr: DCGM_FI_DEV_POWER_USAGE
– record: ai_gpu:temperature_celsius
expr: DCGM_FI_DEV_GPU_TEMP
The example assumes the listed exporter aliases exist in the deployed release. Validate the active collector CSV, exporter output, units, labels, and field support on each GPU generation before promoting these rules. The stable contract for dashboards should be the normalized ai_gpu:* series, not an unreviewed dependency on every upstream field name.
SM Activity Is Not Business Productivity
A high SM activity ratio indicates that streaming multiprocessors were active during the sample window. It does not prove that kernels were efficient, that memory access was optimal, that the model met latency targets, or that the work produced a successful business result.
Low activity can indicate idle reservation, input starvation, CPU or storage bottlenecks, small batches, synchronization, or an interactive workload waiting for a user. High activity can coexist with low requests per second, excessive time to first token, retries, or failed requests.
Sampling Can Miss Short Work
GPU telemetry is sampled. Short kernels and short-lived processes may start and finish between samples. Process-level attribution therefore has a confidence boundary even when the platform exposes process metrics. Increase collection frequency only after measuring the storage, CPU, and cardinality cost. High-frequency raw metrics may belong in short retention, while aggregated evidence belongs in long retention.
Kubernetes Attribution from Device Assignment to Tenant
Kubernetes attribution requires several control-plane facts to line up:
A device plugin or Dynamic Resource Allocation driver advertises GPU resources [6].
The scheduler selects a node based on the resource request, policy, topology, and available capacity.
The kubelet assigns device identifiers to containers.
The local PodResources API exposes container and exclusive-resource assignments to monitoring agents [5].
DCGM Exporter maps device telemetry to pod and container metadata where the allocation mode and release support that mapping.
Kubernetes metadata is joined to approved tenant, project, service, and cost-center identifiers.
Allocation events are retained after the pod disappears.
Use Pod UID and Controller UID, Not Pod Name Alone
A deployment may create many pods with changing names. A job may be retried. An operator may recreate a model-serving pod under the same logical service. The ownership record should therefore preserve:
namespace UID
pod UID
container ID
owning controller UID and kind
workload revision
node
allocated device UUID or MIG identity
assignment start and end time
The human-readable namespace, deployment, and pod names remain useful labels, but the immutable identifiers are the evidence keys.
Create a Bounded Identity Metric
One practical pattern is a small metadata exporter that emits a gauge with value 1 for each active workload identity. The gauge carries only allowlisted, bounded labels.
ai_workload_identity{
cluster=”prod-ai-1″,
namespace=”risk-models”,
pod=”fraud-llm-7c9d8″,
pod_uid=”4d31…”,
workload_uid=”7a4c…”,
tenant_id=”finance”,
project_id=”fraud-platform”,
service_tier=”production”
} 1
Recording rules can then enrich normalized GPU series:
groups:
– name: ai-gpu-tenant-attribution
rules:
– record: ai_tenant_gpu:sm_active_ratio
expr: |
ai_gpu:sm_active_ratio
* on (cluster, namespace, pod)
group_left(tenant_id, project_id, service_tier, pod_uid, workload_uid)
ai_workload_identity
– record: ai_tenant_gpu:framebuffer_used_bytes
expr: |
ai_gpu:framebuffer_used_bytes
* on (cluster, namespace, pod)
group_left(tenant_id, project_id, service_tier, pod_uid, workload_uid)
ai_workload_identity
This design gives the platform team a deliberate schema and a testable failure mode. When a GPU series cannot join to an identity series, it becomes an unattributed telemetry incident instead of silently landing in the wrong tenant report.
Queue Time Has Multiple Meanings
A single panel named “queue time” is not enough.
Queue or delay
Start
End
What it reveals
Admission delay
Workload submitted
Workload admitted
Policy, quota, approval, or admission-control pressure
Scheduler delay
Pod created or job queued
Node assignment
Capacity shortage, topology constraints, fairness, priority, or fragmentation
Startup delay
Node assignment
Container or endpoint ready
Image pull, model load, storage, initialization, or health-check delay
Model-server queue
Request accepted
Request scheduled for execution
Concurrency, batching, KV cache, runtime, or GPU saturation
End-to-end latency
Client request
Final response
Full service experience, including network and application layers
Capacity forecasting should treat sustained scheduler delay as unmet demand. Model-server queue delay is a service bottleneck and may occur even when Kubernetes has already allocated the GPU.
MIG Attribution Is Stronger Than Time-Slicing, but the Math Still Matters
MIG creates hardware-isolated GPU instances with identifiable profiles and instance identifiers. This makes allocation attribution stronger than time-slicing because the platform can associate a specific MIG device with a pod or VM.
The billing unit should be the allocated MIG profile over time:
MIG allocation cost
= MIG-profile-hours x approved profile rate
Do not bill a MIG tenant from the parent GPU utilization percentage. Different instance profiles have different capacity denominators, and parent-device activity is not necessarily a simple sum or average of instance percentages.
Capacity-Normalized Utilization
For fleet planning, the platform may calculate a GPU-equivalent active measure:
Capacity-normalized active equivalent
= sum for each MIG instance of
profile capacity share x instance active ratio
The profile capacity share must come from an approved profile inventory and validation method. It should not be guessed from framebuffer memory alone. Memory, compute slices, media engines, and supported metrics do not always scale identically. Community questions involving heterogeneous MIG layouts also demonstrate why a simple weighted reconstruction of parent utilization should not be treated as authoritative without validation [23].
Keep three distinct views:
Physical GPU view: health, total power, thermals, parent-device activity, and failure domain.
MIG allocation view: profile, assigned tenant, profile-hours, and stranded profile capacity.
MIG workload view: instance activity, memory, model metrics, queue time, and productive work.
Fragmentation Is Its Own Capacity State
A GPU may have free aggregate capacity but still be unable to satisfy a requested MIG profile. That is fragmentation, not ordinary utilization. Capacity dashboards should report:
unallocated physical GPU equivalents
available profiles by GPU model
requested profiles waiting in queue
capacity trapped in incompatible layouts
reconfiguration opportunities and disruption cost
Without this view, a platform can report low average utilization while users experience long queue times.
Time-Slicing Attribution Requires an Explicit Confidence Boundary
Time-slicing lets multiple workloads share a physical GPU through scheduling rather than hard partitioning. It can improve accessibility for development and bursty work, but it complicates attribution.
A physical-device metric may be repeated across time-slice replicas because the underlying measurement belongs to the GPU, not to each logical replica. Dividing the physical utilization equally among configured replicas is not exact attribution. It is a modeled allocation assumption.
Recent DCGM Exporter releases added per-process GPU metrics for time-sharing and MIG, and newer releases continue to improve Kubernetes and container-runtime mapping [4]. That does not make process attribution universal. Hardware support, driver support, DCGM support, exporter configuration, process-accounting support, container-runtime mapping, and sampling behavior still determine what evidence is available.
Process-Level Attribution Chain
A process-level design needs to resolve:
GPU process sample
-> host PID
-> container cgroup or runtime container ID
-> pod UID or VM UUID
-> model-server worker
-> tenant, project, model, and endpoint
Failure can occur at every step. PIDs are reused. Processes terminate between samples. Multi-process services fork workers. NVIDIA MPS can change process behavior. A shared model server may intentionally serve several tenants from one process. In that last case, process telemetry can attribute activity to the service but not accurately divide it among customers. Request and token records must perform the customer split.
Attribution Confidence Levels
Confidence
Typical allocation mode
Evidence quality
Appropriate use
High
Exclusive GPU, MIG device, dedicated vGPU
Stable device-to-workload mapping
Chargeback, SLO analysis, audit evidence
Medium
Supported process mapping on shared GPU
Sampled PID-to-container mapping with known gaps
Showback, efficiency analysis, provisional allocation
Modeled
Time-slice replicas without complete process evidence
Policy-based proportional split or equal-share assumption
Capacity planning with disclosure, not exact billing
Unattributed
Missing metadata, unsupported device, telemetry gap
No defensible mapping
Platform exception queue and cost suspense account
A community report involving Blackwell GB10 is a useful warning. The visible symptom looked like missing per-process attribution, but the vendor response identified the platform as unsupported by DCGM [22]. The operational lesson is to validate the exact GPU and software support boundary before treating exporter output as proof of a general product limitation.
vGPU and VM Attribution Needs Host and Guest Evidence
Virtualized GPU environments need two connected views.
The host or hypervisor view establishes the physical GPU, vGPU instance, vGPU profile, VM UUID, and assignment interval. NVIDIA’s management interfaces can expose vGPU-instance and vGPU-process utilization where supported [16], [17]. The guest view provides telemetry scoped to the VM and, when applicable, the model server or application running inside it.
Recommended vGPU Evidence Chain
Physical GPU UUID
-> vGPU UUID and profile
-> hypervisor VM UUID
-> VM owner and cost center
-> guest workload or model endpoint
-> requests, tokens, latency, and outcomes
Use vGPU-profile-hours for allocation cost. Use vGPU activity for efficiency. Use guest model metrics for productive work.
Watch the Utilization Denominator
A utilization percentage reported inside a vGPU can be relative to the vGPU’s permitted engine capacity, depending on the scheduler. A VM that fully uses its allocated share may report 100 percent even though it is consuming only part of the physical GPU. Host and guest percentages therefore need documented semantics before they are placed on the same chart.
Avoid Double Counting
If host-side vGPU telemetry and guest-side DCGM or NVML telemetry describe the same activity, do not sum them. Use one as the authoritative allocation and device view, and the other as a reconciliation and service detail view.
Pass-through devices require another boundary. The vGPU management layer may not manage pass-through GPUs from the hypervisor, so guest telemetry and hypervisor assignment records become the primary evidence sources.
Model-Server Metrics Convert GPU Activity into Productive Work
Device telemetry explains hardware behavior. Model-server telemetry explains service behavior.
Triton Inference Server exposes request counts, inference counts, execution counts, request timing, and GPU statistics. Its inference and execution counters can also reveal effective batching behavior [12]. vLLM exposes request, token, queue, latency, KV cache, and optional model-flops metrics [13], [14]. NVIDIA NIM for large language models provides Prometheus-compatible metrics and, for supported runtime paths, surfaces runtime-native metrics rather than inventing a separate semantic layer [15].
Core Inference Metric Definitions
Metric
Definition
Why it matters
Requests per second
Rate of accepted, successful, failed, or completed request counters
Demand, throughput, errors, service capacity
Input tokens per second
Rate of prompt-token counters
Prefill workload and input intensity
Output tokens per second
Rate of generated-token counters
Decode throughput and productive output
Time to first token
Request timing to first generated token, using the runtime’s documented start point
User-perceived responsiveness and prefill pressure
Inter-token latency
Time between output tokens during decode
Streaming quality and decode consistency
End-to-end latency
Request arrival to completion
Full service experience
Model queue time
Time waiting in the runtime scheduler before execution
Saturation, batching, concurrency, KV cache pressure
Running requests
Requests actively executing or in an execution batch
Concurrency and active load
Waiting requests
Requests queued for execution
Backlog and unmet service demand
KV cache usage
Fraction or bytes of key-value cache occupied
Concurrency limit, sequence pressure, eviction risk
Average dynamic batch size
Inferences divided by executions when semantics support the calculation
Batching efficiency and throughput behavior
Success rate
Successful requests divided by total completed requests
Productive output quality gate
Per-model latency
Latency histogram segmented by model and deployment revision
Model-specific SLO and regression analysis
Token and Request Rates
Use counters and calculate rates in the query layer:
sum by (tenant_id, model) (
rate(ai_workload:output_tokens_total[5m])
)
sum by (tenant_id, model) (
rate(ai_workload:requests_total{outcome=”success”}[5m])
)
For latency, use histograms and agreed percentile windows:
histogram_quantile(
0.95,
sum by (le, tenant_id, model) (
rate(ai_workload:ttft_seconds_bucket[10m])
)
)
Do Not Put Request IDs in Prometheus
Per-request metrics are useful, but they should normally be returned in the response, written to structured logs, or attached to distributed traces. A request ID as a Prometheus label creates an unbounded series dimension.
A practical split is:
Prometheus: model, endpoint, tenant class, project, outcome, latency bucket, and bounded service dimensions.
Logs or traces: request ID, caller identity, prompt and output token counts, exact model revision, queue time, TTFT, retry chain, and failure detail.
Financial aggregation: daily or hourly tenant totals produced from immutable request events.
Shared Model Servers Need Request-Level Tenant Accounting
A single model-server process can serve multiple tenants. Device metrics and process metrics can attribute GPU activity to the server, but not necessarily to each customer. For this pattern, customer showback should be based on request and token evidence, then reconcile against the service’s allocated GPU cost.
One useful approach is to calculate a tenant’s share of a shared endpoint using an approved workload driver:
Tenant service share
= tenant weighted productive units
/ total endpoint weighted productive units
Weighted units may account for input tokens, output tokens, image size, model class, priority tier, or validated compute coefficients. The coefficient model must be versioned and tested. A raw output-token count alone may not fairly represent prefill-heavy or multimodal workloads.
Normalize Metrics Before Building Dashboards or Bills
Raw sources expose different names, units, labels, and update intervals. A platform-wide semantic layer prevents every dashboard from inventing its own interpretation.
A useful canonical contract includes:
Canonical series
Type
Required bounded dimensions
Purpose
ai_gpu:allocation_fraction
Gauge or interval record
cluster, tenant, project, workload, GPU class, allocation class
Assigned capacity
ai_gpu:sm_active_ratio
Gauge
cluster, node, GPU or instance identity
Normalized activity
ai_gpu:framebuffer_used_bytes
Gauge
cluster, device identity
Memory occupancy
ai_gpu:power_watts
Gauge
cluster, physical GPU
Power observation
ai_gpu:health_incidents_total
Counter
cluster, physical GPU, category, severity
Reliability evidence
ai_scheduler:queue_seconds
Histogram
cluster, queue, tenant, priority, GPU class
Scheduling delay
ai_workload:requests_total
Counter
tenant, project, model, endpoint, outcome
Service demand and success
ai_workload:input_tokens_total
Counter
tenant, model, endpoint
Prefill volume
ai_workload:output_tokens_total
Counter
tenant, model, endpoint
Generated output
ai_workload:ttft_seconds
Histogram
tenant class, model, endpoint
Responsiveness
ai_workload:e2e_latency_seconds
Histogram
tenant class, model, endpoint, outcome
Service-level evidence
ai_finance:gpu_rate_per_hour
Versioned ledger dimension
GPU class, profile, service tier, period
Cost calculation input
Prometheus should hold operational series. The financial rate record should normally live in a controlled data store and be exported into the reporting model, not edited casually as a dashboard constant.
Showback and Chargeback Need Different Governance
Showback reports consumption and cost without creating a financial transfer. Chargeback posts or reallocates cost to a tenant, department, or customer. The telemetry can be shared, but the governance bar is different.
Showback Can Tolerate Provisional Data
Showback can include near-real-time allocation, current utilization, idle reservations, queue delay, estimated cost, and attribution-confidence flags. It is useful for behavior change and capacity conversations.
Chargeback Requires a Closed Ledger
Chargeback should use a closed period with:
approved rate-card version
finalized allocation intervals
late telemetry and correction policy
unassigned-cost treatment
guaranteed and borrowed capacity rules
shared-overhead allocation rule
credits and service-level adjustments
finance and platform approval
immutable monthly output
Guaranteed Quota Versus Borrowed Usage
A fair scheduler may let a tenant exceed guaranteed quota when other tenants are not using their entitlement [18], [19]. That creates at least two distinct capacity classes:
Guaranteed capacity: capacity the platform promises to make available under defined conditions.
Borrowed capacity: temporary over-quota capacity that can be reclaimed or preempted according to policy.
Report them separately:
Guaranteed GPU-hours
Borrowed GPU-hours
Preempted borrowed GPU-hours
Idle guaranteed GPU-hours
Productive units delivered by each capacity class
A common financial model charges guaranteed quota at a predictable committed rate and borrowed allocation at a burst rate. Another model charges only actual allocation but reports unused entitlement as a planning inefficiency. The correct choice is organizational policy, not a telemetry fact.
Classify Idle Capacity Correctly
Idle class
Description
Likely financial owner
Tenant-reserved idle
Guaranteed or dedicated capacity assigned but inactive
Tenant, unless centrally subsidized
Borrowed idle
Over-quota capacity assigned but no longer useful
Tenant until reclaimed, with aggressive scheduler cleanup
Platform unallocated idle
Capacity available to the shared pool
Platform or central AI service
Fragmentation idle
Capacity unusable because of MIG profile, topology, memory, or scheduling constraints
Platform, or shared with the tenant when caused by a dedicated design choice
Reliability reserve
Capacity deliberately held for failure, maintenance, or recovery
Platform overhead
Telemetry-unknown idle
Capacity appears idle because evidence is missing
Suspense category pending investigation
Calling all idle GPU time “waste” produces the wrong behavior. Cost frameworks such as OpenCost also distinguish shared, allocated, and idle cost treatment rather than collapsing them into one figure [20]. Reliability reserve is intentional. Tenant-reserved idle may be the cost of an availability promise. Fragmentation may require layout changes, not user coaching.
Example Grafana Dashboards
A useful dashboard system provides different views for platform operators, tenant owners, model-service teams, and finance. One giant dashboard will satisfy none of them.
Fleet Capacity and Health Dashboard
Primary audience: platform operations and hardware teams.
Recommended variables:
cluster
site
GPU model
node pool
node
physical GPU UUID
MIG mode
health severity
Recommended panels:
installed, allocatable, allocated, unavailable, and unassigned GPU equivalents
current MIG layouts and free profiles
GPU memory used and reserved
SM activity distribution
power and temperature by node and rack
thermal or power clock events
XID, ECC, row-remap, PCIe, and NVLink health signals where supported
telemetry freshness and missing exporter targets
top nodes by fragmentation and queue pressure
Tenant Showback Dashboard
Primary audience: department, customer, project, and service owners.
Recommended variables:
billing period
tenant
department
project
namespace or VM group
service tier
allocation class
GPU or vGPU profile
Recommended panels:
guaranteed, borrowed, and total allocated GPU-hours
idle allocated GPU-hours
active-equivalent GPU-hours
requests, tokens, jobs, or other productive units
cost by guaranteed, borrowed, variable, and overhead component
cost per successful request, token, training job, or approved outcome
queue time and preemption history
attribution-confidence breakdown
top idle workloads with owner and age
Model Efficiency and SLO Dashboard
Primary audience: model platform and application teams.
Recommended panels:
requests per second
input and output tokens per second
p50, p95, and p99 TTFT
inter-token latency and end-to-end latency
runtime queue depth and queue time
running and waiting requests
KV cache usage
GPU memory and SM activity aligned to service metrics
failures, retries, cancellations, and SLO compliance
cost per successful request and cost per million output tokens
model, quantization, runtime, and deployment-revision comparison
Scheduler and Capacity Dashboard
Primary audience: platform engineering, capacity planning, and service management.
Recommended panels:
pending GPU workloads by tenant, priority, GPU class, and age
scheduler delay percentiles
guaranteed quota versus borrowed consumption
preemption and reclaim events
available capacity that cannot satisfy queued requests
MIG and topology fragmentation
requested versus delivered GPU profiles
forecast peak by week and month
failure-domain reserve and maintenance headroom
Dashboard Wireframe
Grafana variables should make the dashboards reusable instead of duplicating one dashboard per tenant. Provision data sources, folders, dashboards, and alert integrations through version-controlled configuration [10], [11]. Apply role-based access so one customer cannot query another customer’s telemetry.
Alert Examples That Protect Evidence and Capacity
Alerts should identify actionable conditions, not merely display high values. Route hardware-health alerts to platform operations, tenant-efficiency alerts to service owners, and billing-evidence alerts to the telemetry team.
The following example uses normalized metrics. Thresholds are placeholders and must be validated by GPU model, workload class, service level, and operating policy.
groups:
– name: aias-gpu-accountability
rules:
– alert: GPUAllocationMissingTenantIdentity
expr: |
ai_gpu:allocation_fraction > 0
unless on (cluster, workload_uid)
ai_workload_identity == 1
for: 10m
labels:
severity: critical
owner: telemetry-platform
annotations:
summary: Allocated GPU capacity has no tenant identity
action: Quarantine the interval from chargeback and repair the mapping
– alert: TenantGPUAllocatedButIdle
expr: |
avg_over_time(ai_tenant_gpu:allocation_fraction[30m]) > 0
and on (cluster, tenant_id, workload_uid)
avg_over_time(ai_tenant_gpu:sm_active_ratio[30m]) < 0.05
for: 30m
labels:
severity: warning
owner: tenant-platform
annotations:
summary: Tenant workload holds GPU capacity with minimal activity
action: Validate workload intent before reclaiming or resizing capacity
– alert: GPUHealthIncidentDetected
expr: |
increase(ai_gpu:health_incidents_total{severity=~”critical|fatal”}[5m]) > 0
labels:
severity: critical
owner: gpu-operations
annotations:
summary: Critical GPU health incident detected
action: Preserve evidence, evaluate workload evacuation, and follow the hardware runbook
– alert: InferenceQueueSLOAtRisk
expr: |
histogram_quantile(
0.95,
sum by (le, cluster, endpoint, model) (
rate(ai_workload:queue_seconds_bucket[10m])
)
) > 30
for: 10m
labels:
severity: warning
owner: model-platform
annotations:
summary: Model-server queue delay threatens the service objective
action: Check concurrency, batching, KV cache, replicas, and GPU saturation
– alert: BorrowedGPUCapacityApproachingPolicyLimit
expr: |
ai_tenant_gpu:borrowed_allocation_fraction
/ ai_tenant_gpu:borrow_limit_fraction > 0.9
for: 15m
labels:
severity: warning
owner: capacity-management
annotations:
summary: Tenant borrowed GPU consumption is near its policy limit
action: Review reclaim risk, queued demand, and production priority
Alertmanager should group related device failures by cluster and node, inhibit secondary tenant alerts when a node is already declared unavailable, and route financial-evidence failures differently from performance warnings [9]. Missing attribution is a critical data-quality alert because it can invalidate chargeback even when workloads continue running.
Capacity Forecasting Requires More Than Average Utilization
Average GPU utilization hides peaks, queue pressure, model mix, profile fragmentation, and failure reserve. Forecast at least four demand curves:
Committed demand: guaranteed quota and dedicated allocations promised to tenants.
Allocated demand: observed concurrent GPU equivalents, MIG profiles, and vGPU profiles.
Active demand: capacity-normalized device activity during allocated periods.
Productive demand: requests, tokens, training throughput, or other service outcomes.
Add a fifth curve for unmet demand, derived from pending workloads, scheduler delay, admission delay, and model-server queues.
Forecast by Capacity Class
Do not treat every GPU as interchangeable. Segment forecasts by:
GPU model and memory size
MIG profile
vGPU profile
interconnect and topology requirement
security zone
production, development, or batch service tier
model family, precision, and runtime
whole-GPU, MIG, time-slice, or vGPU operating mode
Use Validated Service Throughput
For an inference service, estimate capacity from a controlled benchmark that preserves the target SLO:
Required service GPUs
= forecast peak productive throughput
/ validated sustainable throughput per GPU at the target SLO
+ failure and maintenance reserve
The validated denominator must match model, precision, runtime, batch policy, sequence distribution, concurrency, and hardware. A tokens-per-second number obtained under a different latency target is not a safe capacity input.
Queue Pressure Is Demand Evidence
When utilization looks moderate but queue time rises, investigate fragmentation, memory reservation, topology, admission policy, model loading, and runtime bottlenecks. Queue pressure can expose demand that utilization alone cannot see.
Evidence Retention and Auditability
Prometheus is excellent for operational metrics, but local Prometheus storage alone is not a financial evidence system. Long-term evidence needs durable retention, controlled schemas, and reproducible monthly close.
A practical design can use Prometheus for local collection and alerting, then remote-write or upload blocks into a durable Prometheus-compatible backend backed by object storage [21]. Request and trace events can use a separate analytical store. Monthly cost output should be materialized into an immutable ledger or governed warehouse table.
Example Retention Tiers
These are planning examples, not universal legal requirements:
Evidence
Example retention
Reason
High-resolution device and process metrics
30 to 90 days
Incident response, tuning, short-term attribution investigation
Five-minute operational rollups
13 to 25 months
Seasonality, trend, capacity, budget comparison
Allocation events and ownership history
At least the financial and audit horizon
Reconstructs who controlled capacity
Request and token aggregates
Based on customer, security, and finance policy
Productive-work showback and service economics
Monthly chargeback ledger
Commonly aligned to finance record-retention policy
Audit, reconciliation, dispute handling
Raw prompts and outputs
Only when required, minimized and protected
High privacy and data-governance risk
Preserve the Calculation Context
Each closed period should retain:
metric schema version
normalization-rule version
rate-card version
tenant and project hierarchy snapshot
allocation-class definitions
source-system versions
data-gap and correction records
attribution-confidence distribution
total platform cost and reconciliation result
approver and close timestamp
A future reviewer should be able to reproduce the charge without relying on the current state of Kubernetes, Grafana, or the hypervisor.
Implementation Sequence
A platform team can build this architecture incrementally, but the sequence matters.
Phase
Key work
Exit evidence
Identity contract
Define tenant, project, service, device, workload, and finance keys
Approved schema and ownership policy
Device inventory
Record physical GPU, MIG, vGPU, node, VM, and capacity-class relationships
Reconciled inventory with stable identifiers
Device telemetry
Deploy DCGM Exporter or supported VM telemetry and validate fields
Known-good metrics by GPU generation and allocation mode
Allocation capture
Persist Kubernetes, scheduler, quota, VM, and vGPU assignment intervals
No orphan allocation intervals in test cases
Metadata enrichment
Join device evidence to workload and business identities
Attribution confidence and exception reporting
Productive-work telemetry
Instrument Triton, vLLM, NIM, training, or application outcomes
Request, token, latency, queue, and success evidence
Semantic normalization
Create canonical metrics and recording rules
Versioned metric contract and unit tests
Showback pilot
Produce tenant dashboards and estimated cost
Tenant review and dispute findings
Chargeback readiness
Add rate cards, close controls, reconciliation, and corrections
Finance-approved monthly close process
Forecasting and optimization
Add peak, queue, profile, failure-reserve, and outcome forecasts
Capacity plan tied to service demand and SLOs
Start with Showback
Showback exposes data-quality problems without immediately creating financial disputes. Use the pilot to discover missing labels, pod-lifecycle gaps, ambiguous shared services, unsupported GPU modes, and confusing rate policies. Promote only high-confidence allocation classes into chargeback.
Validation Tests Before Financial Use
A telemetry architecture should be tested like a production control, not accepted because a dashboard displays data.
Exclusive GPU Test
Run one known pod or VM on one exclusive GPU. Confirm:
the physical GPU UUID is correct
assignment start and end times match the scheduler or hypervisor
tenant and project mapping is correct
memory, SM activity, power, and model metrics move together
allocated GPU-hours reconcile to elapsed time
MIG Test
Create different MIG profiles and run controlled workloads. Confirm:
each device maps to the correct profile and tenant
instance metrics remain separate
the parent GPU is not double counted
capacity-normalized reporting uses approved profile weights
reconfiguration and fragmentation events are visible
Time-Slicing Test
Run two workloads with distinct load patterns. Confirm whether the deployed stack provides process-level evidence. Compare observed attribution with controlled request or job totals. Mark the result as modeled when the system cannot isolate activity reliably.
vGPU Test
Map the physical GPU, vGPU UUID, profile, VM UUID, guest telemetry, and model endpoint. Confirm host and guest percentages use documented denominators and are not summed.
Shared Model-Service Test
Send requests from multiple tenants through one process. Confirm that request and token events divide customer consumption while device and process metrics remain assigned to the shared service. Reconcile tenant service shares to the endpoint’s total cost.
Lifecycle and Failure Tests
Test pod deletion, pod rescheduling, VM reassignment, exporter restart, Prometheus outage, clock skew, model revision change, node failure, telemetry gaps, and late-arriving data. A month-end process that works only during steady state is not ready for chargeback.
Reconciliation Controls
Use explicit equations:
Physical capacity hours
= allocated hours
+ unallocated hours
+ unavailable or maintenance hours
+ approved reconciliation difference
Recoverable platform cost
= tenant charges
+ central platform charges
+ approved subsidies
– credits
+ approved reconciliation difference
Set a small tolerated difference, investigate exceptions, and prevent the period from closing when unattributed capacity or cost exceeds policy.
Operational Guardrails
A mature AIaaS telemetry service needs operating ownership.
Platform operations own GPU health, node telemetry, exporter availability, and hardware incident response.
Kubernetes or virtualization teams own allocation mappings and scheduler or hypervisor evidence.
Model platform teams own request, token, queue, latency, and model-revision telemetry.
FinOps or finance owns approved rates, overhead policy, and financial close.
Tenant owners review idle reservations, service outcomes, and forecast demand.
Security and privacy teams review customer identifiers, trace content, retention, and dashboard access.
Document the support boundary for every GPU generation, driver, DCGM release, exporter release, vGPU release, Kubernetes version, runtime, and hypervisor integration. A metric that works on one validated cluster should not be assumed to work on every device or virtualization path.
Conclusion
The question “who used the GPU?” has no reliable one-metric answer.
DCGM and DCGM Exporter can show what a supported GPU was doing. Kubernetes PodResources, scheduler evidence, device allocation records, and hypervisor inventory can show which workload controlled the device. Triton, vLLM, NIM, training frameworks, and application telemetry can show whether the workload produced useful service outcomes. A governed cost ledger can then apply guaranteed-quota, borrowed-capacity, variable-service, and shared-overhead rules.
The architecture becomes defensible when those measurements remain separate but connected. Allocation answers ownership. Utilization answers hardware activity. Productive work answers service value. Financial consumption answers policy-based cost responsibility.
That separation also makes the difficult cases manageable. MIG can be billed by profile-hours while utilization is analyzed per instance. vGPU can be billed by profile assignment while guest and host telemetry are reconciled. Time-slicing can carry a modeled confidence flag instead of an invented exact percentage. Shared model services can allocate customer cost through request and token evidence rather than trying to split a single process metric.
The platform team’s goal is not to produce the most colorful GPU dashboard. It is to create an evidence chain that survives pod deletion, process churn, node failure, model upgrades, tenant disputes, capacity reviews, and month-end close. When that evidence chain exists, GPU observability becomes more than monitoring. It becomes the operating record for AI-as-a-Service.
External References
[1] NVIDIA: DCGM ExporterCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html
[2] NVIDIA: Feature Overview, NVIDIA DCGM DocumentationCanonical URL: https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
[3] NVIDIA: Field Identifiers, NVIDIA DCGM DocumentationCanonical URL: https://docs.nvidia.com/datacenter/dcgm/latest/dcgm-api/dcgm-api-field-ids.html
[4] NVIDIA: NVIDIA/dcgm-exporter ReleasesCanonical URL: https://github.com/NVIDIA/dcgm-exporter/releases
[5] Kubernetes: Kubernetes 1.28: Node podresources API Graduates to GACanonical URL: https://kubernetes.io/blog/2023/08/23/kubelet-podresources-api-ga/
[6] Kubernetes: Device PluginsCanonical URL: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
[7] Prometheus: OverviewCanonical URL: https://prometheus.io/docs/introduction/overview/
[8] Prometheus: Metric and Label NamingCanonical URL: https://prometheus.io/docs/practices/naming/
[9] Prometheus: AlertmanagerCanonical URL: https://prometheus.io/docs/alerting/latest/alertmanager/
[10] Grafana Labs: Prometheus Template VariablesCanonical URL: https://grafana.com/docs/grafana/latest/datasources/prometheus/template-variables/
[11] Grafana Labs: Provision GrafanaCanonical URL: https://grafana.com/docs/grafana/latest/administration/provisioning/
[12] NVIDIA: Metrics, NVIDIA Triton Inference ServerCanonical URL: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/metrics.html
[13] vLLM Project: Production MetricsCanonical URL: https://docs.vllm.ai/en/v0.18.1/usage/metrics/
[14] vLLM Project: Per-Request MetricsCanonical URL: https://docs.vllm.ai/en/latest/features/per_request_metrics/
[15] NVIDIA: Logging and Observability, NVIDIA NIM for Large Language ModelsCanonical URL: https://docs.nvidia.com/nim/large-language-models/latest/reference/logging-and-observability.html
[16] NVIDIA: Management SDK User GuideCanonical URL: https://docs.nvidia.com/vgpu/latest/grid-management-sdk-user-guide/index.html
[17] NVIDIA: NVML API Reference Guide, vGPU Utilization and Accounting APIsCanonical URL: https://docs.nvidia.com/deploy/nvml-api/group__nvmlUtil.html
[18] NVIDIA Run:ai: Over Quota, Fairness and PreemptionCanonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/scheduling/quick-starts/over-quota
[19] NVIDIA Run:ai: The NVIDIA Run:ai Scheduler, Concepts and PrinciplesCanonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/scheduling/concepts-and-principles
[20] OpenCost: OpenCost SpecificationCanonical URL: https://opencost.io/docs/specification/
[21] Grafana Labs: Introduction to Grafana MimirCanonical URL: https://grafana.com/docs/mimir/latest/introduction/
[22] NVIDIA Developer Forums: DCGM-Exporter, Missing Process-Level Attribution for GPU Time-Slicing on Blackwell GB10Canonical URL: https://forums.developer.nvidia.com/t/dcgm-exporter-missing-process-level-attribution-for-gpu-time-slicing-on-blackwell-gb10/363037
[23] NVIDIA Developer Forums: MIG Instances Utilization CalculationCanonical URL: https://forums.developer.nvidia.com/t/mig-instances-utilization-calculation/341194
[24] Hacker News: Show HN, Neurox, GPU Observability for AI InfraCanonical URL: https://news.ycombinator.com/item?id=43835948
Why Your GPU Is Idle: A Layer by Layer Troubleshooting Guide for Enterprise Inference
Introduction An enterprise inference service can look busy while its GPU remains nearly idle. The application may be accepting requests, retrieving documents,…
Next PostGPU Multi-Tenancy Without Security Theater: Isolation, Quotas, Noisy Neighbors, and Confidential ComputingIntroduction GPU sharing is easy to describe and difficult to govern. A platform team can expose one physical accelerator as several scheduler-visible resources, divide it into Multi-Instance GPU partitions, present…
The post Who Used the GPU? Building Per Tenant Telemetry, Showback, and Capacity Evidence for AIaaS appeared first on Digital Thought Disruption.