TL;DR
The NVIDIA GPU Operator automates the software stack required to make GPUs usable by Kubernetes workloads. It can deploy and manage NVIDIA drivers, the NVIDIA Container Toolkit, the Kubernetes device plugin, GPU Feature Discovery, DCGM Exporter, MIG Manager, and validation components.
A successful installation requires more than running a Helm command. The GPU hardware must be visible to the operating system, the Kubernetes and Linux versions must be supported, the driver ownership model must be decided, the container runtime must be compatible, and the device plugin must successfully advertise nvidia.com/gpu resources to Kubernetes.
This guide installs a pinned GPU Operator release, validates the node software stack, runs a CUDA workload, and establishes a practical troubleshooting and upgrade workflow.
Introduction
Adding a GPU-equipped server to a Kubernetes cluster does not automatically make the GPU available to containers.
Kubernetes can schedule CPU and memory without vendor-specific software, but GPUs require several additional layers. The Linux operating system needs a compatible NVIDIA driver. The container runtime needs a supported device-injection path. Kubernetes needs a device plugin that can advertise GPU resources to the scheduler. Operators also need monitoring, validation, labeling, and a controlled driver lifecycle.
Installing each component separately creates an operational dependency chain that is difficult to maintain consistently across a cluster. A driver update can break the container runtime. A kernel update can prevent the driver module from loading. A missing device plugin can leave healthy GPUs invisible to the scheduler.
The NVIDIA GPU Operator addresses this problem by managing the GPU software stack through Kubernetes resources and DaemonSets.
This article focuses on a conventional Kubernetes cluster running Linux GPU nodes on bare metal or virtual machines with GPU passthrough. NVIDIA vGPU, OpenShift, managed cloud Kubernetes services, confidential containers, and KubeVirt introduce additional platform-specific requirements that should be validated separately.
What You Will Accomplish
By the end of this tutorial, you will be able to:
Check whether the Kubernetes platform, operating system, and runtime are suitable for the GPU Operator.
Prepare GPU worker nodes for driver installation.
Choose between Operator-managed, host-installed, and precompiled drivers.
Install a pinned GPU Operator release using Helm.
Verify the driver, toolkit, device plugin, GPU discovery, and telemetry components.
Confirm that Kubernetes advertises nvidia.com/gpu resources.
Run a CUDA validation workload.
Plan Operator and driver upgrades.
Troubleshoot pods that cannot access GPUs.
How the GPU Node Software Stack Works
The GPU Operator does not replace Kubernetes scheduling. It prepares the node so that Kubernetes can understand and allocate the GPU.
The important point in the following diagram is that each layer depends on the layer below it. When a workload cannot access a GPU, troubleshooting should start at the hardware and driver layers rather than at the application.
The GPU Operator manages most of these layers through a ClusterPolicy resource. Node Feature Discovery identifies NVIDIA hardware, the driver DaemonSet prepares the kernel, the toolkit configures device injection, and the device plugin exposes schedulable resources.
Check the Supported Platform Before Installation
Do not assume that every combination of Linux, Kubernetes, kernel, runtime, and driver is supported.
For the current 26.3 release line, NVIDIA validates several mainstream Linux distributions against Kubernetes 1.32 through 1.36. The exact range varies by operating system and Kubernetes distribution.
The current matrix also validates containerd 1.7 through 2.2 and CRI-O on supported operating systems. Additional features such as the Node Resource Interface plugin have stricter minimum runtime versions.
Use the following checklist before making changes.
CheckWhat to ConfirmGPU hardwareThe GPU model is supported and visible through PCI or the virtualization platformKubernetesThe cluster version appears in the current GPU Operator platform matrixOperating systemThe GPU node distribution and release are supportedKernelThe running kernel has compatible headers, packages, or precompiled driver supportContainer runtimecontainerd or CRI-O is running at a supported versionDeployment modelBare metal, PCI passthrough, or a supported vGPU designSecurity configurationSecure Boot, SELinux, and Pod Security settings are understoodImage accessNodes can reach required container registries and operating system repositoriesExisting softwareDriver and toolkit ownership is known before the Operator is installed
Run these basic cluster checks from an administrative workstation:
kubectl version
kubectl get nodes -o wide
kubectl get nodes -o custom-columns=NAME:.metadata.name,OS:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion
On each intended GPU node, verify that the hardware is visible:
lspci -nn | grep -i NVIDIA
uname -r
If lspci does not show the GPU, the GPU Operator cannot fix the problem. Investigate BIOS configuration, PCI passthrough, hypervisor assignment, cloud instance configuration, or physical hardware installation first.
Prepare the GPU Worker Nodes
Node preparation is where many GPU Operator installations succeed or fail.
Stabilize the Kernel Lifecycle
A driver container must build, load, or provide kernel modules that match the running Linux kernel. Uncontrolled kernel updates can therefore break a previously healthy GPU node.
Check the running kernel:
uname -r
For Ubuntu nodes, confirm that matching kernel headers are available:
dpkg -l | grep “linux-headers-$(uname -r)” || true
For RHEL-compatible nodes, check the matching development package:
rpm -q “kernel-devel-$(uname -r)” || true
If the packages are not installed, make sure the node can retrieve them from an approved operating system repository. In restricted environments, mirror the required packages or use a validated precompiled driver strategy.
Do not allow an automated kernel update policy to move GPU nodes to a kernel that has not been tested with the selected driver branch.
Check for Existing Drivers
Determine whether the host already has an NVIDIA driver:
lsmod | grep -E ‘^nvidia’ || true
command -v nvidia-smi >/dev/null && nvidia-smi || true
This decision affects the Helm configuration:
No host driver: allow the GPU Operator to deploy the driver.
Host driver already installed: disable the Operator-managed driver.
Driver and NVIDIA Container Toolkit already installed: disable both components in the Operator.
Immutable or tightly controlled nodes: consider precompiled drivers or a baked node image.
Do not intentionally install two competing driver-management systems on the same node.
Check the Nouveau Module
The open-source nouveau driver can conflict with NVIDIA driver initialization, particularly in vGPU environments.
Check whether it is loaded:
lsmod | grep nouveau || true
If it is present, follow the operating system and NVIDIA guidance for the selected deployment model. Denylisting the module normally requires updating the initramfs and rebooting the node.
Check Secure Boot
Inspect the current state when mokutil is available:
mokutil –sb-state 2>/dev/null || true
The current GPU Operator troubleshooting guidance identifies EFI Secure Boot as unsupported for the normal Operator-managed driver path and instructs administrators to disable it. Treat this as a platform design gate rather than discovering it during installation.
Verify Container Runtime Health
For containerd:
containerd –version
sudo systemctl status containerd –no-pager
For CRI-O:
crio –version
sudo systemctl status crio –no-pager
Also confirm that kubelet is healthy:
sudo systemctl status kubelet –no-pager
A GPU Operator installation cannot compensate for an unhealthy runtime, missing CRI socket, or kubelet configuration problem.
Confirm Registry and Package Access
A default installation pulls several container images. The driver container might also need operating system packages while preparing the node.
Validate that the GPU nodes have:
DNS resolution.
Outbound access to the required NVIDIA registry endpoints.
Access to operating system package repositories.
Access to any internal registry mirror.
Trusted certificate chains for private repositories.
Working image pull secrets when private images are used.
Air-gapped clusters require a separate mirroring and repository design. Do not begin with the default connected installation and assume it can later be converted without additional work.
Choose the Driver Installation Model
The driver model is the most important configuration decision in the installation.
ModelBest FitOperator ConfigurationOperational OwnershipOperator-managed driverStandard Kubernetes GPU node poolsdriver.enabled=trueGPU Operator manages the containerized driverPreinstalled host driverGolden images, managed cloud nodes, DGX-style imagesdriver.enabled=falseNode image or host-management processPreinstalled driver and toolkitFully prepared GPU node imageDisable driver and toolkitNode image and operating system teamPrecompiled driver containerControlled kernels and supported precompiled combinationsEnable precompiled modeOperator plus kernel compatibility processCustom driver imageSpecialized or isolated environmentsCustom repository and versionPlatform team, with limited vendor support
Operator-Managed Containerized Drivers
This is the default installation model.
The Operator deploys a driver DaemonSet to each detected GPU worker node. It installs and loads the appropriate NVIDIA kernel modules and makes the driver filesystem available to the other GPU components.
This model provides the most integrated lifecycle, but it also means the Operator needs privileged access and close interaction with the host kernel.
Preinstalled Host Drivers
Use host-installed drivers when the node image or infrastructure platform already manages the driver.
The installation override is:
–set driver.enabled=false
The GPU Operator can still manage the toolkit, device plugin, monitoring, GPU discovery, and other operands.
The Operator does not manage the lifecycle of a driver that was installed directly on the host. Driver upgrades must remain part of the node image, operating system, or infrastructure lifecycle.
Preinstalled Driver and Container Toolkit
When both components are already installed, use:
–set driver.enabled=false
–set toolkit.enabled=false
NVIDIA documentation also requires the existing runtime configuration to be prepared correctly before using this model. Validate that the host toolkit and container runtime configuration match the Kubernetes workload pattern.
Precompiled Drivers
Precompiled driver containers avoid building kernel modules during node initialization. They can reduce installation dependencies and improve predictability, but only for supported operating system, architecture, kernel, and driver-branch combinations.
The relevant Helm settings are:
–set driver.usePrecompiled=true
–set driver.version=”<supported-driver-branch>”
Do not use an arbitrary branch number. Select one from the component and precompiled-driver matrices for the exact Operator release and kernel.
Open or Proprietary Kernel Modules
Current charts use driver.kernelModuleType=auto by default. With supported modern driver branches, the Operator selects the recommended open or proprietary kernel module based on the GPU and driver.
The available values are:
auto
open
proprietary
The older driver.useOpenKernelModules option is deprecated. Do not copy that setting from an outdated installation guide.
Understand the Current Container Runtime Model
Beginning with GPU Operator 25.10, Container Device Interface support is enabled by default.
CDI provides a standardized method for describing and injecting devices into containers. For normal Kubernetes workloads that request GPUs through the NVIDIA device plugin, CDI is largely transparent.
A standard GPU pod normally needs only:
resources:
limits:
nvidia.com/gpu: 1
It does not need runtimeClassName: nvidia simply to consume a GPU.
This is an important difference from many older GPU Operator tutorials.
CDI and Standard GPU Workloads
The device plugin allocates the GPU, and the container runtime uses CDI to inject the device, libraries, and required mounts.
With the default configuration:
cdi:
enabled: true
Application teams should request GPUs through Kubernetes resources rather than manually setting NVIDIA_VISIBLE_DEVICES.
Optional NRI Integration
The Node Resource Interface plugin can handle device injection without modifying the container runtime configuration.
Enable it with:
cdi:
enabled: true
nriPluginEnabled: true
NRI requires a compatible containerd or CRI-O release. Validate the current minimum versions before enabling it.
For a first installation, leave NRI disabled unless you have a specific runtime-management requirement and have tested it with the Kubernetes distribution.
Custom containerd Paths
The default toolkit configuration expects the conventional containerd configuration and socket paths:
/etc/containerd/config.toml
/run/containerd/containerd.sock
K3s, RKE2, custom packages, and some managed distributions use different paths. In those environments, set the toolkit environment values or use the platform-specific installation instructions.
Do not overwrite a managed runtime configuration without understanding which service regenerates it.
Install the GPU Operator with Helm
The commands below use GPU Operator v26.3.3, which is the current patch release at the time of writing.
Pinning the version makes the installation reproducible. Recheck the release notes and support matrix before using a newer chart.
Set the Installation Variables
Set the chart version and the NVIDIA Helm repository endpoint. The repository endpoint is listed in External References.
export GPU_OPERATOR_VERSION=”v26.3.3″
export NVIDIA_HELM_REPOSITORY=”<NVIDIA Helm repository endpoint>”
Create the Namespace
The GPU Operator requires privileged workloads because it installs drivers, mounts host paths, and modifies runtime configuration.
Create the namespace:
kubectl create namespace gpu-operator
–dry-run=client
-o yaml | kubectl apply -f –
If Pod Security Admission is enforced, label the namespace:
kubectl label –overwrite namespace gpu-operator
pod-security.kubernetes.io/enforce=privileged
Review this exception with the Kubernetes security team. The privileged namespace should be restricted to the Operator and its managed components.
Add the Helm Repository
helm repo add nvidia “${NVIDIA_HELM_REPOSITORY}”
helm repo update nvidia
Confirm the chart is available:
helm search repo nvidia/gpu-operator –versions | head
Create a Baseline Values File
Create gpu-operator-values.yaml:
driver:
enabled: true
kernelModuleType: auto
toolkit:
enabled: true
cdi:
enabled: true
nriPluginEnabled: false
nfd:
enabled: true
dcgmExporter:
enabled: true
This values file makes the key decisions visible without unnecessarily overriding every chart default.
Change the following only when required:
Set driver.enabled to false for host-installed drivers.
Set toolkit.enabled to false for a host-installed toolkit.
Set nfd.enabled to false if the cluster already runs a compatible Node Feature Discovery deployment.
Enable NRI only after runtime compatibility has been verified.
Pin driver.version when your support or validation process requires a specific driver.
Configure image pull secrets when using private repositories.
Inspect all available settings before installation:
helm show values nvidia/gpu-operator
–version “${GPU_OPERATOR_VERSION}”
> gpu-operator-default-values.yaml
Install the Chart
helm install gpu-operator nvidia/gpu-operator
–namespace gpu-operator
–version “${GPU_OPERATOR_VERSION}”
–values gpu-operator-values.yaml
–wait
–timeout 20m
The –wait flag causes Helm to wait for the release resources, but it does not replace the component-level validation that follows.
Check the Helm release:
helm list -n gpu-operator
helm status gpu-operator -n gpu-operator
Verify the Operator Components
A default installation normally deploys components for:
NVIDIA GPU driver.
NVIDIA Container Toolkit.
NVIDIA Kubernetes Device Plugin.
GPU Feature Discovery.
Node Feature Discovery.
DCGM Exporter.
MIG Manager on compatible systems.
Operator validation.
List the pods:
kubectl get pods -n gpu-operator -o wide
Healthy long-running pods should show Running. One-time CUDA validation pods can show Completed.
Check the ClusterPolicy:
kubectl get clusterpolicy
The expected state is ready.
Inspect the full resource if it is not ready:
kubectl describe clusterpolicy cluster-policy
Check the managed DaemonSets:
kubectl get daemonsets -n gpu-operator
For each GPU node, the desired and ready counts should align for the driver, toolkit, device plugin, feature discovery, and telemetry DaemonSets that apply to that node.
Confirm Device Plugin Registration and GPU Discovery
The device plugin registers GPUs with kubelet as an extended resource named nvidia.com/gpu.
Check node capacity and allocatable resources:
kubectl get nodes
-o custom-columns=NAME:.metadata.name,GPUS:.status.allocatable.nvidia.com/gpu
A healthy GPU node should show the expected number of allocatable GPUs.
Inspect a specific node:
kubectl describe node <gpu-node-name>
Look for a capacity section similar to:
Capacity:
nvidia.com/gpu: 1
Allocatable:
nvidia.com/gpu: 1
GPU Feature Discovery also adds useful labels that can support scheduling and inventory.
Review them with:
kubectl get node <gpu-node-name> –show-labels
For a cleaner view:
kubectl get node <gpu-node-name> -o json
| jq ‘.metadata.labels | with_entries(select(.key | startswith(“nvidia.com/”)))’
Common labels describe properties such as:
GPU presence.
GPU product.
Driver version.
CUDA capability.
GPU count.
MIG capability or configuration.
GPU memory.
These labels can later support node affinity, admission policies, chargeback, or workload-placement rules.
Run a CUDA Validation Workload
Resource advertisement proves that the scheduler can see the GPU. It does not prove that a container can execute a CUDA kernel.
Create cuda-vectoradd.yaml:
apiVersion: v1
kind: Pod
metadata:
name: cuda-vectoradd
spec:
restartPolicy: Never
containers:
– name: cuda-vectoradd
image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
resources:
limits:
nvidia.com/gpu: 1
Notice that the pod requests one GPU but does not specify runtimeClassName. That is intentional for the current CDI-based default configuration.
Apply the workload:
kubectl apply -f cuda-vectoradd.yaml
Watch its status:
kubectl get pod cuda-vectoradd -w
When the pod completes, inspect the logs:
kubectl logs cuda-vectoradd
A successful test includes:
Test PASSED
Done
Delete the validation pod:
kubectl delete -f cuda-vectoradd.yaml
If your environment cannot pull the public validation image, mirror it into the approved internal registry and update the YAML.
Validate GPU Access Inside a Running Container
A second validation pattern is useful for confirming nvidia-smi output from inside a container.
Create nvidia-smi-test.yaml:
apiVersion: v1
kind: Pod
metadata:
name: nvidia-smi-test
spec:
restartPolicy: Never
containers:
– name: nvidia-smi
image: nvcr.io/nvidia/cuda:12.8.1-base-ubuntu24.04
command:
– nvidia-smi
resources:
limits:
nvidia.com/gpu: 1
Apply it and inspect the logs:
kubectl apply -f nvidia-smi-test.yaml
kubectl wait –for=condition=Ready pod/nvidia-smi-test –timeout=120s || true
kubectl logs nvidia-smi-test
The output should identify the allocated GPU and driver.
Remove the pod:
kubectl delete -f nvidia-smi-test.yaml
Always verify that the CUDA container image is compatible with the selected driver branch before standardizing it for production.
Establish an Operational Baseline
Before onboarding application teams, capture a known-good state.
helm get values gpu-operator
-n gpu-operator
-o yaml > gpu-operator-installed-values.yaml
kubectl get clusterpolicy cluster-policy
-o yaml > cluster-policy-baseline.yaml
kubectl get nodes
-o custom-columns=NAME:.metadata.name,GPUS:.status.allocatable.nvidia.com/gpu
> gpu-capacity-baseline.txt
kubectl get pods -n gpu-operator -o wide
> gpu-operator-pods-baseline.txt
Also record:
GPU Operator chart version.
Driver version.
Container Toolkit version.
Device plugin version.
GPU model and firmware.
Operating system and kernel version.
Kubernetes version.
Container runtime version.
Validation workload image.
Validation result and date.
This baseline becomes important after kernel updates, driver upgrades, node replacements, or cluster upgrades.
Plan Operator Upgrades Carefully
The GPU Operator is a cluster infrastructure component. Treat upgrades as platform changes rather than routine application releases.
The current lifecycle policy supports upgrades within the current major release or to the next major release. If a cluster is several release families behind, do not assume that a direct jump is supported.
Prepare the Upgrade
Before upgrading:
Read the target release notes and component matrix.
Verify the Kubernetes, Linux, kernel, runtime, and GPU combination.
Export the existing Helm values and ClusterPolicy.
Test the change on a representative non-production GPU node pool.
Identify workloads that must be stopped or drained during driver changes.
Confirm rollback boundaries.
Verify image and package availability.
Capture current GPU health and DCGM metrics.
Export the existing configuration:
helm get values gpu-operator
-n gpu-operator
-o yaml > gpu-operator-values-before-upgrade.yaml
kubectl get clusterpolicy cluster-policy
-o yaml > cluster-policy-before-upgrade.yaml
helm history gpu-operator -n gpu-operator
Upgrade the Helm Release
Set the target release:
export TARGET_GPU_OPERATOR_VERSION=”<validated-target-version>”
Refresh the repository:
helm repo update nvidia
Fetch and review the target values:
helm show values nvidia/gpu-operator
–version “${TARGET_GPU_OPERATOR_VERSION}”
> gpu-operator-values-target.yaml
Compare the current and target values instead of blindly reusing an old file:
diff -u gpu-operator-values-before-upgrade.yaml
gpu-operator-values-target.yaml || true
Perform the upgrade:
helm upgrade gpu-operator nvidia/gpu-operator
–namespace gpu-operator
–version “${TARGET_GPU_OPERATOR_VERSION}”
–values gpu-operator-values-target.yaml
–disable-openapi-validation
–wait
–timeout 30m
Current GPU Operator releases use a Helm pre-upgrade hook for CRD upgrades. The hook has been enabled by default since the 24.9 release line.
After the upgrade, repeat the full component, allocatable-resource, CUDA, and telemetry validation.
Understand Driver Upgrades
The upgrade controller is enabled by default and is NVIDIA’s recommended method for upgrading Operator-managed containerized drivers.
A driver upgrade must:
Stop clients that are using the driver.
Unload the current kernel modules.
replace the driver pod.
Install and load the new modules.
Restore GPU clients.
Monitor driver upgrade state:
kubectl get nodes -l nvidia.com/gpu.present
-o jsonpath='{range .items[*]}{.metadata.name}{“t”}{.metadata.labels.nvidia.com/gpu-driver-upgrade-state}{“n”}{end}’
A completed node should eventually report upgrade-done.
Drivers that were installed directly on the host are outside the Operator’s lifecycle. Those nodes must use the operating system or image-management process for upgrades.
Know the Rollback Boundary
A Helm rollback can restore chart-managed Kubernetes resources:
helm history gpu-operator -n gpu-operator
helm rollback gpu-operator <revision>
-n gpu-operator
–wait
–timeout 30m
However, a Helm rollback might not fully reverse:
Updated CRDs.
A host kernel change.
A driver module already installed or loaded.
Runtime configuration changes.
A MIG geometry change.
A node image replacement.
Document the rollback plan at the chart, driver, kernel, runtime, and node-image layers.
Troubleshoot Pods That Cannot See GPUs
Do not start by changing the application image or reinstalling the entire Operator.
Use a bottom-up workflow.
Symptom and Cause Matrix
SymptomLikely LayerFirst ChecksNo NVIDIA PCI deviceHardware or virtualizationlspci, passthrough, instance typeDriver pod stuck in initKernel or driverDriver logs, kernel headers, repositories, Secure Bootnvidia-smi failsDriver or hardwareDriver pod, dmesg, NVRM and Xid messagesNo runtime for nvidiaToolkit or RuntimeClassToolkit logs and runtime configurationNode shows no nvidia.com/gpuDevice plugin or kubeletDevice plugin pod and node capacityPod remains PendingScheduler or capacityPod events, GPU requests, taints, affinityWrong number of GPUsDriver, device plugin, MIG, hardwarenvidia-smi -L, allocatable resources, MIG configCUDA test failsImage and driver compatibilityDriver version, CUDA image requirementsValidator repeatedly failsLower stack dependencyDriver, toolkit, Fabric Manager, or hardware health
Check Operator State First
kubectl get clusterpolicy
kubectl get pods -n gpu-operator -o wide
kubectl get events -n gpu-operator –sort-by=.lastTimestamp
Find components that are not Running or Completed.
Describe the failed pod:
kubectl describe pod <pod-name> -n gpu-operator
Review all containers in the pod:
kubectl logs <pod-name>
-n gpu-operator
–all-containers
–tail=200
Check the Driver Layer
List driver pods and their assigned nodes:
kubectl get pods
-n gpu-operator
-l app=nvidia-driver-daemonset
-o wide
Select the driver pod on the affected node:
kubectl logs <driver-pod>
-n gpu-operator
-c nvidia-driver-ctr
–tail=200
Run nvidia-smi inside the driver container:
kubectl exec
-n gpu-operator
<driver-pod>
-c nvidia-driver-ctr
— nvidia-smi
Check kernel messages:
kubectl exec
-n gpu-operator
<driver-pod>
-c nvidia-driver-ctr
— dmesg | grep -Ei ‘NVRM|Xid|nvidia’
Common driver-layer problems include:
Unsupported kernel.
Missing matching kernel packages.
Secure Boot.
Loaded nouveau module.
Inaccessible operating system repositories.
Incorrect vGPU driver or licensing path.
Hardware errors or GPUs that have fallen off the PCI bus.
Fabric Manager problems on NVSwitch-based systems.
Check the Toolkit and Runtime Layer
List toolkit pods:
kubectl get pods
-n gpu-operator
-l app=nvidia-container-toolkit-daemonset
-o wide
Review the pod on the affected node:
kubectl logs <toolkit-pod>
-n gpu-operator
–all-containers
–tail=200
Inspect the active containerd configuration on the node:
sudo containerd config dump | grep -i -A8 -B4 nvidia || true
For CRI-O:
sudo crio status config | grep -i -A8 -B4 nvidia || true
An error stating that no runtime named nvidia exists usually means a pod explicitly requested the NVIDIA RuntimeClass but the toolkit did not create the corresponding handler.
For ordinary device-plugin workloads on current releases, remove an unnecessary runtimeClassName: nvidia and use a normal nvidia.com/gpu resource limit.
Check the Device Plugin
List the device plugin pods:
kubectl get pods
-n gpu-operator
-l app=nvidia-device-plugin-daemonset
-o wide
Review the affected pod:
kubectl logs <device-plugin-pod>
-n gpu-operator
–all-containers
–tail=200
Check node allocation:
kubectl get node <gpu-node-name>
-o jsonpath='{.status.capacity.nvidia.com/gpu}{” capacity, “}{.status.allocatable.nvidia.com/gpu}{” allocatablen”}’
If the driver works but the resource is missing, investigate:
Device plugin pod failure.
Kubelet device plugin registration.
Node labels.
MIG configuration.
Operator deployment labels.
GPU health.
Runtime or mount failures affecting the device plugin.
Check the Workload Specification
Describe the workload:
kubectl describe pod <workload-pod>
Look at the Events section.
A GPU must normally be requested as a limit:
resources:
limits:
nvidia.com/gpu: 1
Also check:
The requested number of GPUs does not exceed allocatable capacity.
Node affinity allows a GPU node.
Taints have matching tolerations.
The pod is not requesting a MIG resource that the node does not expose.
Namespace quotas do not block the request.
Admission policies have not modified or rejected the pod.
The selected CUDA image supports the installed driver.
Collect a Must-Gather Archive
When component logs do not reveal the cause, use NVIDIA’s GPU Operator must-gather utility according to the current troubleshooting documentation.
The archive collects Kubernetes resources and logs for the Operator and managed operands. Review the contents for sensitive infrastructure information before transferring it to a vendor or external support system.
Practical Production Recommendations
A lab installation can use defaults. A production platform should establish stronger controls.
Pin the GPU Operator chart version.
Record and test the selected driver branch.
Use dedicated GPU node pools.
Control kernel upgrades.
Mirror images for restricted or production environments.
Monitor DCGM metrics and GPU Xid events.
Maintain a representative non-production GPU node.
Run CUDA validation after every platform change.
Define who owns the Operator, node image, kernel, driver, runtime, and workload.
Treat MIG and time slicing as separate capacity-design decisions.
Preserve Helm values, ClusterPolicy, logs, and validation evidence.
Align Kubernetes upgrades with the GPU Operator support matrix.
Do not let application teams bypass Kubernetes allocation by manually injecting devices.
The GPU Operator simplifies installation, but it does not remove the need for platform ownership and lifecycle discipline.
Conclusion
The NVIDIA GPU Operator provides the cleanest general-purpose path for turning Kubernetes GPU nodes into a repeatable platform service.
The installation itself is straightforward once the underlying decisions are made. The hardware must be visible. The operating system, kernel, Kubernetes release, and runtime must be supported. The platform team must choose whether the driver belongs to the Operator or the host image. The toolkit and CDI path must be healthy. Finally, the device plugin must advertise nvidia.com/gpu before Kubernetes can schedule a workload.
The most important operational lesson is to troubleshoot the stack from the bottom up. A missing GPU resource is usually not an application problem. It is evidence that one of the hardware, driver, runtime, toolkit, device plugin, or kubelet layers has not completed its role.
Once this foundation is stable, the cluster is ready for more advanced GPU platform capabilities such as NVIDIA NIM microservices, MIG partitioning, GPU time slicing, DCGM-based monitoring, Kubernetes autoscaling, and governed AI workload deployment.
External References
NVIDIA: Installing the NVIDIA GPU OperatorCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html
NVIDIA: GPU Operator Platform SupportCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/platform-support.html
NVIDIA: GPU Operator Release NotesCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/release-notes.html
NVIDIA: Container Device Interface and Node Resource Interface Plugin SupportCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/cdi.html
NVIDIA: Upgrading the NVIDIA GPU OperatorCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/upgrade.html
NVIDIA: GPU Driver UpgradesCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html
NVIDIA: Troubleshooting the NVIDIA GPU OperatorCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/troubleshooting.html
NVIDIA: Precompiled Driver ContainersCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/precompiled-drivers.html
NVIDIA: GPU Operator Helm RepositoryCanonical URL: https://helm.ngc.nvidia.com/nvidia
Kubernetes: Schedule GPUsCanonical URL: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
Kubernetes: Device PluginsCanonical URL: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
The post How to Install and Configure the NVIDIA GPU Operator on Kubernetes appeared first on Digital Thought Disruption.