TL;DR
NVIDIA NIM can be deployed on Kubernetes through Helm or managed declaratively through the NVIDIA NIM Operator. The operator-based path is the better fit when you want Kubernetes-native lifecycle management for model caching, GPU scheduling, health probes, service exposure, scaling, and upgrades.
The practical sequence is straightforward, but the dependencies matter. Build a supported Kubernetes and GPU foundation, install the GPU Operator, install NIM Operator 3.1.1, create NGC secrets in the workload namespace, populate a persistent NIMCache, and then deploy a NIMService that requests the correct GPU resources and mounts the cache.
The most important production distinction is that NGC credentials authorize image and model downloads. They do not automatically protect the inference endpoint. Client authentication, TLS, rate limiting, and authorization should be enforced at an ingress controller, Kubernetes Gateway API implementation, service mesh, or enterprise API gateway.
Introduction
Running an inference container is easy. Operating a reliable inference service on Kubernetes is not.
The difficult parts are usually outside the container itself: installing and validating GPU drivers, exposing GPU resources to the scheduler, downloading large model artifacts, preserving those artifacts across restarts, matching a model profile to the available GPU, waiting long enough for model initialization, protecting the endpoint, and upgrading without turning every model change into a long outage.
The NVIDIA NIM Operator addresses that operational layer. It extends Kubernetes with custom resources for model caching and NIM service lifecycle management. Instead of manually maintaining a Deployment, Service, persistent volume, health probes, and model-download job, the platform team declares the desired NIM state and allows the operator to reconcile the supporting resources.
This tutorial uses the Automation Tutorial content family because the goal is not simply to explain NIM architecture. The goal is to leave you with an end-to-end implementation path, validation commands, production controls, rollback boundaries, and troubleshooting guidance.
What You Will Accomplish
By the end of this tutorial, you will have:
validated a Kubernetes cluster with schedulable NVIDIA GPU resources
installed the NVIDIA GPU Operator and NVIDIA NIM Operator with pinned versions
created NGC credentials for container and model access
created a persistent model cache with a NIMCache custom resource
deployed a GPU-backed NIMService
tested the first inference endpoint through local port forwarding
understood how to expose the endpoint through Ingress or Gateway API
configured a practical path for health checks, logs, metrics, and horizontal scaling
established upgrade and rollback procedures that preserve the previous model cache
The example uses an NVIDIA-documented LLM-specific NIM image and a single-GPU deployment. Confirm that the selected image tag, model profile, GPU type, memory capacity, and NVIDIA AI Enterprise entitlement are valid for your environment before applying the manifests.
How the NIM Operator Fits into the Request Flow
The application does not send inference traffic through the NIM Operator. The operator is a control-plane component. It watches NIMCache and NIMService resources, then creates and maintains the Kubernetes objects that serve requests.
The runtime request path goes through the Kubernetes Service or external gateway directly to the NIM pod and GPU.
The separation matters operationally. If the operator restarts, an already running NIM pod can continue serving traffic. If the NIM pod fails, Kubernetes and the operator reconcile the declared service state. If the model cache is missing or incompatible, the serving pod may start but never become ready.
Prerequisites and Support Boundaries
The commands in this article use a version baseline validated on July 22, 2026. NVIDIA NIM, NIM Operator, GPU Operator, model images, and supported Kubernetes releases are version-sensitive. Recheck the NVIDIA support matrix and release notes before production deployment.
RequirementPractical baselineWhy it mattersKubernetesVersion 1.26 or later for the documented NIM LLM operator pathOlder API behavior and unsupported platforms can block deployment or supportabilityAdministrative accessCluster-admin for operator installationThe operators install CRDs, controllers, RBAC, and node-level GPU componentsHelmHelm 3Used to install and upgrade both operatorsNVIDIA GPU nodesGPU type and count supported by the selected NIM model profileA pod can request a GPU and still fail if the model does not fit or the profile is incompatibleGPU OperatorVersion 24.3.0 or later, with the article pin set to v26.3.3Provides drivers, container runtime integration, device plugin behavior, runtime classes, validation, and node labelsNIM OperatorVersion 3.1.1Current documented operator release at the research baselineNGC accessNVIDIA AI Enterprise subscription or eligible NVIDIA Developer Program access, plus an NGC API keyNIM images and model artifacts are access controlledPersistent storageA working storage class sized for the selected model profilesModel caches can be tens or hundreds of gigabytesNetwork accessOutbound access to NVIDIA registries and model services, or a validated air-gapped mirrorCache jobs and image pulls fail without the required egress pathEndpoint controlIngress, Gateway API, service mesh, or API gateway for production exposureKubernetes Service exposure alone does not provide enterprise authentication and authorization
Confirm the GPU and Model Fit Before Installation
Do not choose a NIM image solely because the model name looks appropriate. Confirm:
supported GPU families
required GPU count
minimum GPU memory
supported inference engine
expected model cache size
whether the model uses an optimized profile, a community engine, or a buildable profile
whether your NGC organization is entitled to the image and model artifacts
For a first deployment, start with one model, one replica, one supported GPU, and persistent storage. Prove the complete path before adding autoscaling, multiple GPU types, multi-node inference, or shared platform tenancy.
Prepare the Deployment Variables
The NVIDIA Helm repository address is intentionally stored in an environment variable so the article body remains free of publication links. Copy the repository endpoint from the NVIDIA NIM Operator installation guide listed in the External References section.
export NVIDIA_HELM_REPOSITORY=”<NVIDIA_NGC_HELM_REPOSITORY>”
export GPU_OPERATOR_VERSION=”v26.3.3″
export NIM_OPERATOR_VERSION=”3.1.1″
export NIM_NAMESPACE=”nim-service”
export NIM_NAME=”meta-llama-3-2-1b-instruct”
Change the storage class, NIM image, and NIM tag in the manifests before deployment. The image and tag shown in this tutorial come from NVIDIA’s current NIM Operator examples, but they should be treated as a documented implementation example, not a guarantee that the image is entitled or appropriate for every GPU.
Validate the Kubernetes and GPU Foundation
Start by confirming that the cluster, Helm client, and GPU nodes are visible.
kubectl version
helm version
kubectl get nodes -o wide
kubectl get storageclass
If the cluster already has the GPU Operator, inspect the existing installation before creating another release.
helm list -A | grep -i gpu-operator || true
kubectl get pods -A | grep -E ‘nvidia|gpu-operator’ || true
kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count
The nvidia.com/gpu extended resource must appear as allocatable on at least one worker node. This command displays allocatable GPU counts without requiring a test workload:
kubectl get nodes
-o custom-columns=’NODE:.metadata.name,GPU:.status.allocatable.nvidia.com/gpu,GPU_PRODUCT:.metadata.labels.nvidia.com/gpu.product’
A blank GPU column means the Kubernetes scheduler does not currently see an allocatable NVIDIA GPU on that node. Fix the GPU foundation before installing NIM.
Install the NVIDIA GPU Operator
Add the NVIDIA Helm repository and install the GPU Operator with a pinned version.
helm repo add nvidia “${NVIDIA_HELM_REPOSITORY}”
helm repo update
helm upgrade –install gpu-operator nvidia/gpu-operator
–namespace gpu-operator
–create-namespace
–version “${GPU_OPERATOR_VERSION}”
–wait
This default path is appropriate when the GPU Operator should manage the driver and container toolkit. Managed Kubernetes services, DGX systems, preinstalled drivers, vGPU environments, Secure Boot systems, and cloud-specific node images can require different Helm values. Do not apply the default command blindly to a cluster where drivers or the NVIDIA Container Toolkit are already managed by the platform.
Validate the GPU Operator
kubectl get pods -n gpu-operator
kubectl get pods -n gpu-operator -l app=nvidia-cuda-validator
kubectl get runtimeclass
kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count
Expected results include:
the GPU Operator controller is running
driver, toolkit, device plugin, and validation components are healthy where applicable
the CUDA validator completes successfully
an NVIDIA runtime class is present when required by the platform
GPU nodes have NVIDIA feature and product labels
nvidia.com/gpu appears in node allocatable resources
Do not proceed while the CUDA validator is failing or GPU resources are missing from node capacity.
Install the NVIDIA NIM Operator
Create the operator namespace and install the pinned NIM Operator release.
kubectl create namespace nim-operator –dry-run=client -o yaml | kubectl apply -f –
helm upgrade –install nim-operator nvidia/k8s-nim-operator
–namespace nim-operator
–version “${NIM_OPERATOR_VERSION}”
–wait
Validate the controller and CRDs.
kubectl get pods -n nim-operator
kubectl get crd | grep apps.nvidia.com
kubectl api-resources | grep -E ‘NIMCache|NIMService|NIMPipeline’
At minimum, the cluster should recognize the NIMCache and NIMService resource types before you continue.
Configure NGC Authentication and Image Access
Create a dedicated namespace for the NIM workload.
kubectl create namespace “${NIM_NAMESPACE}” –dry-run=client -o yaml | kubectl apply -f –
Export the NGC API key in your shell without writing it into a manifest or source repository.
read -s -p “NGC API key: ” NGC_API_KEY
export NGC_API_KEY
printf ‘n’
Create two secrets in the workload namespace:
ngc-secret is a Docker registry secret used to pull NIM container images
ngc-api-secret is a generic secret used by model puller containers to download model artifacts
kubectl create secret docker-registry ngc-secret
–namespace “${NIM_NAMESPACE}”
–docker-server=nvcr.io
–docker-username=’$oauthtoken’
–docker-password=”${NGC_API_KEY}”
–dry-run=client -o yaml | kubectl apply -f –
kubectl create secret generic ngc-api-secret
–namespace “${NIM_NAMESPACE}”
–from-literal=NGC_API_KEY=”${NGC_API_KEY}”
–dry-run=client -o yaml | kubectl apply -f –
unset NGC_API_KEY
Validate only metadata. Do not print or decode the secret value in routine troubleshooting output.
kubectl get secret -n “${NIM_NAMESPACE}” ngc-secret ngc-api-secret
NGC Authentication Is Not Endpoint Authentication
The two secrets above authenticate image and model downloads. The authSecret field in a NIMService points to the secret containing NGC_API_KEY. It should not be interpreted as client authentication for application requests.
For production, protect the inference endpoint with controls such as:
TLS termination
OIDC or JWT validation
mTLS between trusted workloads
API keys issued and validated by an API gateway
namespace and network policy restrictions
rate limiting and request-size controls
audit logging tied to application or workload identity
Keep the NIM Service as ClusterIP unless there is a specific reason to expose it directly.
Design the Model Cache Before Deploying the Service
Model caching is not a cosmetic optimization. It controls startup time, network demand, repeatability, scaling behavior, and rollback safety.
The NIMCache resource starts a Kubernetes Job that downloads compatible model profiles to persistent storage. A later NIMService mounts that cache instead of downloading the complete model every time a serving pod starts.
Choose the Access Mode Deliberately
For a single-replica lab, ReadWriteOnce is often sufficient. For production scaling across multiple GPU nodes, evaluate whether the storage system and access mode allow every replica to mount the cache when scheduled on different nodes.
Use these rules:
use persistent storage for production
avoid emptyDir for production model persistence because data disappears with the pod
use hostPath only for tightly controlled single-node or proof-of-concept deployments
size storage for the filtered model profiles plus upgrade headroom
use profile filtering so the cache job does not download unnecessary engines or GPU variants
keep the old cache until the upgraded service is fully validated
understand the deletion behavior before allowing the operator to create and own the PVC
When spec.storage.pvc.create is true, deleting the NIMCache can delete the operator-created PVC and cached models. That behavior is convenient for cleanup and dangerous during a rushed rollback.
Create the Persistent NIM Cache
Create a file named nim-cache.yaml.
apiVersion: apps.nvidia.com/v1alpha1
kind: NIMCache
metadata:
name: meta-llama-3-2-1b-instruct
namespace: nim-service
spec:
source:
ngc:
modelPuller: nvcr.io/nim/meta/llama-3.2-1b-instruct:1.12.0
pullSecret: ngc-secret
authSecret: ngc-api-secret
model:
engine: tensorrt_llm
tensorParallelism: “1”
storage:
pvc:
create: true
storageClass: “<YOUR_STORAGE_CLASS>”
size: “80Gi”
volumeAccessMode: ReadWriteOnce
Change the storage class and validate that the engine and tensor parallelism match a profile supported by your GPU. If the model supports several engines and you do not filter correctly, the cache can consume significantly more storage than expected.
Apply the cache resource.
kubectl apply -f nim-cache.yaml
kubectl get nimcache -n “${NIM_NAMESPACE}” -w
The cache must reach Ready before the serving workload is deployed.
Inspect the supporting PVC, Job, events, and logs when progress stalls.
kubectl get pvc -n “${NIM_NAMESPACE}”
kubectl get jobs -n “${NIM_NAMESPACE}”
kubectl describe nimcache -n “${NIM_NAMESPACE}” “${NIM_NAME}”
kubectl logs -n “${NIM_NAMESPACE}” -l job-name=”${NIM_NAME}-job” –tail=200
A successful cache stage proves four things at once: NGC authentication works, the image is entitled, the cluster can reach the required registries, and persistent storage can hold the selected model profile.
Deploy the First NIM Endpoint
Create a file named nim-service.yaml.
apiVersion: apps.nvidia.com/v1alpha1
kind: NIMService
metadata:
name: meta-llama-3-2-1b-instruct
namespace: nim-service
labels:
app.kubernetes.io/name: meta-llama-3-2-1b-instruct
app.kubernetes.io/part-of: nvidia-nim
spec:
labels:
app.kubernetes.io/name: meta-llama-3-2-1b-instruct
app.kubernetes.io/part-of: nvidia-nim
image:
repository: nvcr.io/nim/meta/llama-3.2-1b-instruct
tag: “1.12.0”
pullPolicy: IfNotPresent
pullSecrets:
– ngc-secret
authSecret: ngc-api-secret
storage:
nimCache:
name: meta-llama-3-2-1b-instruct
profile: “”
replicas: 1
resources:
limits:
nvidia.com/gpu: 1
expose:
service:
type: ClusterIP
port: 8000
The blank profile allows NIM to select a compatible cached profile. Pin a specific profile only after you have inspected the cached profile list and validated it against the target GPU.
Apply the service.
kubectl apply -f nim-service.yaml
kubectl get nimservice -n “${NIM_NAMESPACE}” -w
Inspect the generated Kubernetes objects.
kubectl get deployment,pod,service -n “${NIM_NAMESPACE}”
kubectl describe nimservice -n “${NIM_NAMESPACE}” “${NIM_NAME}”
kubectl get events -n “${NIM_NAMESPACE}” –sort-by=.lastTimestamp | tail -n 30
A large model can take several minutes to initialize even when the cache is ready. Do not treat a temporarily failing readiness check as a container crash without checking startup progress and logs.
Test the NIM Endpoint
Use port forwarding for the first validation. This keeps the service internal and removes ingress, DNS, certificates, and gateway policies from the initial test.
kubectl port-forward -n “${NIM_NAMESPACE}”
service/”${NIM_NAME}” 8000:8000
In another terminal, check the health endpoints.
curl -s localhost:8000/v1/health/live
curl -s localhost:8000/v1/health/ready
Discover the model name exposed by the service.
curl -s localhost:8000/v1/models | jq
Send a small chat completion request using the model identifier returned by the models endpoint.
curl -s localhost:8000/v1/chat/completions
-H ‘Content-Type: application/json’
-d ‘{
“model”: “meta/llama-3.2-1b-instruct”,
“messages”: [
{
“role”: “user”,
“content”: “Explain why persistent model caching matters in Kubernetes.”
}
],
“max_tokens”: 128
}’ | jq
Successful execution should produce a valid JSON response and a generated assistant message. A successful liveness check with a failing readiness check usually means the container is running but the model has not finished loading.
Expose the Service Without Bypassing Security
The safest default is ClusterIP. Add an external path only after the internal service is healthy.
Exposure methodBest fitPrimary caveatPort forwardingAdministrator validation and troubleshootingNot an application access patternClusterIPInternal applications in the same cluster or connected service networkRequires internal identity and network policy designIngressExisting HTTP ingress standard with TLS and authentication integrationsController-specific annotations and behavior must be governedGateway APIPlatform teams standardizing routes, gateways, policy attachment, and cross-namespace ownershipRequires a Gateway API implementation and pre-created GatewayLoadBalancerSimple external connectivity in a controlled environmentCan create a direct entry point that bypasses shared gateway controls
Example Ingress-Based Exposure
The NIM Operator can generate an Ingress resource through the spec.expose.router.ingress field. Add the following structure to the NIMService specification after your ingress controller, DNS, and TLS secret are ready.
spec:
expose:
service:
type: ClusterIP
port: 8000
router:
ingress:
ingressClass: nginx
tlsSecretName: nim-api-tls
hostDomainName: ai.example.com
The generated hostname follows the operator naming pattern using the NIM service name, namespace, and host domain.
Authentication is still a gateway responsibility. Configure the ingress controller or upstream API gateway to validate client identity before traffic reaches the NIM service. Do not place a public LoadBalancer in front of the service and assume the ngc-api-secret protects it.
Restrict East-West Access with NetworkPolicy
This example permits traffic only from namespaces explicitly labeled for NIM access.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-approved-nim-clients
namespace: nim-service
spec:
podSelector:
matchLabels:
app.kubernetes.io/part-of: nvidia-nim
policyTypes:
– Ingress
ingress:
– from:
– namespaceSelector:
matchLabels:
access.nim/enabled: “true”
ports:
– protocol: TCP
port: 8000
Label only approved client namespaces.
kubectl label namespace <CLIENT_NAMESPACE> access.nim/enabled=true
NetworkPolicy support depends on the cluster network plugin. Validate enforcement rather than assuming the policy is active.
Understand the Default Health Checks
The NIM Operator configures three health behaviors by default:
ProbeDefault endpointOperational meaningLiveness/v1/health/liveConfirms the container and proxy are alive; repeated failure restarts the podReadiness/v1/health/readyControls whether the service sends traffic to the podStartup/v1/health/readyProtects slow model initialization from premature liveness and readiness actions
The documented startup probe permits up to approximately 20 minutes by default through a 10-second period and failure threshold of 120. Large models, slow storage, engine construction, or cold caches can require a longer window.
Do not disable probes simply to make a deployment look healthy. Increase the startup window only when logs show legitimate model initialization that needs more time.
spec:
startupProbe:
enabled: true
probe:
httpGet:
path: /v1/health/ready
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 240
A forty-minute startup allowance can hide genuine failures if it is applied without monitoring. Pair longer startup thresholds with cache validation, log inspection, and deployment time objectives.
Configure Logging and Operational Visibility
Start with Kubernetes-native inspection.
kubectl logs -n “${NIM_NAMESPACE}”
-l app=”${NIM_NAME}”
–all-containers=true
–tail=200
kubectl logs -n nim-operator
-l app.kubernetes.io/name=k8s-nim-operator
–tail=200
The operator-managed NIM LLM path uses JSONL logging and an INFO log level by default. Increase verbosity temporarily when diagnosing a deployment, then return to the normal level to avoid excessive volume and sensitive request context.
spec:
env:
– name: NIM_LOG_LEVEL
value: DEBUG
Production logging should separate:
operator reconciliation logs
model cache Job logs
NIM application logs
ingress or gateway access logs
GPU telemetry
Kubernetes events
authentication and authorization decisions
Do not rely on pod logs alone for a service-level view. Correlate requests, queue depth, latency, errors, GPU utilization, pod readiness, cache state, and gateway identity.
Enable Metrics and Horizontal Scaling
NIM LLM exposes inference metrics, and the operator can create a Prometheus ServiceMonitor. The cluster must already have the Prometheus Operator CRDs and a matching discovery label.
spec:
metrics:
enabled: true
serviceMonitor:
additionalLabels:
release: kube-prometheus-stack
interval: 30s
scrapeTimeout: 10s
Horizontal scaling should use inference-specific pressure rather than generic CPU utilization. GPU inference pods can have high GPU saturation while CPU remains moderate, or queue requests while memory and CPU metrics appear acceptable.
The following example scales on a custom metric representing waiting requests.
spec:
scale:
enabled: true
hpa:
minReplicas: 1
maxReplicas: 4
metrics:
– type: Pods
pods:
metric:
name: vllm:num_requests_waiting
target:
type: AverageValue
averageValue: “10”
When spec.scale.enabled is true, remove spec.replicas from the NIMService. The HPA owns the replica count.
This configuration also requires a Prometheus Adapter or equivalent custom metrics bridge. Verify that the metric appears in the Kubernetes custom metrics API before enabling production autoscaling.
Scaling Is Also a Capacity and Storage Decision
Before increasing maxReplicas, confirm:
enough schedulable GPUs exist for the maximum replica count
the model cache can be mounted by replicas across the intended nodes
the gateway can distribute requests correctly
startup time is compatible with the expected traffic ramp
the cluster autoscaler can add compatible GPU nodes when required
GPU quotas and priority classes prevent lower-value workloads from consuming serving capacity
the model license and NVIDIA AI Enterprise subscription cover the deployment
An HPA that requests four replicas does not create four GPUs. It only creates pending pods when capacity is unavailable.
Upgrade the NIM Operator Safely
Upgrade the operator separately from the model-serving workload. The operator can be upgraded while NIM custom resources are running, but CRD handling requires attention because Helm does not automatically upgrade existing CRDs in the normal chart lifecycle.
Export the current values and manifests before changing the operator.
helm get values nim-operator -n nim-operator -o yaml > nim-operator-values-before.yaml
kubectl get nimcache,nimservice -A -o yaml > nim-resources-before.yaml
Update the chart repository and inspect the target values.
helm repo update nvidia
helm show values nvidia/k8s-nim-operator
–version “${NIM_OPERATOR_VERSION}”
> nim-operator-values-target.yaml
Perform the upgrade with your reviewed values file.
helm upgrade nim-operator nvidia/k8s-nim-operator
–namespace nim-operator
–version “${NIM_OPERATOR_VERSION}”
-f nim-operator-values-target.yaml
Validate the controller, CRDs, and existing custom resources after the upgrade.
kubectl rollout status deployment -n nim-operator –timeout=5m
kubectl get pods -n nim-operator
kubectl get nimcache,nimservice -A
If the chart upgrade fails, inspect the pre-upgrade hook and image pull events before retrying. A failed hook can leave the Helm release in a failed state even while existing NIM services continue running.
Upgrade a NIM Model Without Destroying the Rollback Point
A model image upgrade is not only a Deployment image change. New NIM versions can introduce different model profiles, cache artifacts, engine behavior, and storage requirements.
Use this sequence:
Create a new NIMCache with a new name and the target NIM image tag.
Wait for the new cache to reach Ready.
Export the current NIMService manifest.
Update the NIM image tag and cache reference together.
Keep at least two replicas when availability during a rolling update is required and supported by available GPU capacity.
Validate health, model discovery, inference output, logs, latency, and error rate.
Retain the previous cache and manifest until the rollback window closes.
A practical naming pattern is:
meta-llama-3-2-1b-instruct-cache-1-12-0
meta-llama-3-2-1b-instruct-cache-<NEXT_VERSION>
The serving resource can retain a stable service name while the cache name changes between releases.
Roll Back Predictably
Rollback should restore a known combination of image tag, cache, environment variables, probe settings, and routing configuration.
NIM Workload Rollback
kubectl apply -f nim-service-previous.yaml
kubectl get nimservice -n “${NIM_NAMESPACE}” -w
kubectl rollout status deployment -n “${NIM_NAMESPACE}” –timeout=30m
Validate the old endpoint before deleting the failed cache.
NIM Operator Rollback
List Helm revisions.
helm history nim-operator -n nim-operator
Roll back to a known release revision.
helm rollback nim-operator <REVISION> -n nim-operator –wait
A Helm rollback does not guarantee that CRDs are restored to an earlier schema. Review CRD compatibility before downgrading the operator. CRD changes are one reason operator rollback should be tested in a non-production cluster.
Common Deployment Failures
SymptomLikely causeWhat to checkImagePullBackOffInvalid NGC key, missing entitlement, wrong image tag, registry egress blockedPod events, secret namespace, image name, NGC organization access, proxy and firewall rulesNIMCache remains PendingPVC cannot bind or cache Job cannot scheduleStorageClass, PVC events, quota, node selector, access modeNIMCache becomes NotReadyCache Job failed, disk full, profile filter mismatch, authentication or network failureCache Job logs, NIMCache status conditions, PVC capacity, NGC secretsNIM pod remains PendingNo compatible GPU is allocatable, taints are not tolerated, node selector is wrong, RWO volume cannot attachPod events, GPU node labels, quotas, taints, storage topologyNIM pod enters CrashLoopBackOffIncompatible model profile, driver or runtime issue, missing cache content, GPU memory exhaustionPod logs, GPU Operator validation, cache readiness, model support matrixLiveness passes but readiness failsModel is still loading or initialization failedStartup logs, readiness endpoint, startup probe window, cache mountEndpoint works through port forwarding but not externallyIngress or Gateway route, DNS, TLS, NetworkPolicy, or authentication configuration is wrongGenerated route, gateway listener, certificate, service port, policy logsHPA shows unknown metricsServiceMonitor not discovered or custom metrics adapter is missingPrometheus targets, ServiceMonitor labels, custom metrics API, metric nameNIM resources are not reconciledOperator pod, RBAC, webhook, or CRD issueOperator logs, CRDs, events, Helm release statusNew replica cannot mount the cacheStorage access mode or topology does not support multi-node mountsPVC access mode, CSI driver behavior, pod node placement, RWX availability
Fast Diagnostic Command Set
kubectl get nimcache,nimservice -n “${NIM_NAMESPACE}”
kubectl get pod,pvc,job,service -n “${NIM_NAMESPACE}”
kubectl get events -n “${NIM_NAMESPACE}” –sort-by=.lastTimestamp | tail -n 50
kubectl describe nimcache -n “${NIM_NAMESPACE}” “${NIM_NAME}”
kubectl describe nimservice -n “${NIM_NAMESPACE}” “${NIM_NAME}”
kubectl logs -n “${NIM_NAMESPACE}” -l job-name=”${NIM_NAME}-job” –tail=200
kubectl logs -n “${NIM_NAMESPACE}” -l app=”${NIM_NAME}” –all-containers=true –tail=200
kubectl logs -n nim-operator -l app.kubernetes.io/name=k8s-nim-operator –tail=200
The order matters. Start with custom resource status, then inspect the Kubernetes objects created by the operator. Avoid changing three layers at once because that destroys the evidence needed to identify the original failure.
Production Readiness Checklist
Before directing production traffic to the endpoint, confirm:
the model image, tag, profile, GPU type, and GPU count are documented
the GPU Operator and NIM Operator versions are pinned
operator values and custom resources are stored in version control
NGC secrets are managed through the organization’s secret-management process
Kubernetes secrets at rest are protected according to platform policy
the model cache is persistent, sized, monitored, and backed by a defined lifecycle policy
the external route terminates TLS and validates client identity
NetworkPolicy or equivalent segmentation limits east-west access
request and token limits are enforced at the appropriate layer
startup, readiness, and liveness behavior are tested under cold-start conditions
logs, metrics, GPU telemetry, gateway identity, and Kubernetes events are centralized
HPA metrics are visible before autoscaling is enabled
maximum replicas align with available GPU capacity and storage access mode
the previous model cache and manifest remain available during the rollback window
an owner exists for platform operations, model lifecycle, endpoint security, and application integration
Conclusion
The NIM Operator turns NVIDIA NIM deployment into a Kubernetes-native operating model rather than a collection of manually maintained resources.
The clean implementation pattern is to establish the GPU foundation first, install the operator control plane second, populate persistent model storage third, and deploy the inference service only after the cache is ready. That sequence makes failures easier to isolate because each stage proves a specific dependency before the next layer is introduced.
The most important production decisions sit around the NIM pod, not inside it. Storage topology determines whether replicas can scale across nodes. Gateway design determines whether the endpoint is authenticated and governed. Probe settings determine whether slow model initialization is tolerated without hiding genuine failures. Upgrade discipline determines whether a model change is reversible.
For a first deployment, keep the design intentionally small: one documented NIM image, one compatible GPU, one persistent cache, one ClusterIP service, and one controlled test client. Once that path is repeatable, add external routing, observability, autoscaling, higher availability, and multi-model operations as deliberate platform capabilities.
External References
NVIDIA: Installing NVIDIA NIM OperatorCanonical URL: https://docs.nvidia.com/nim-operator/latest/install.html
NVIDIA: Release NotesCanonical URL: https://docs.nvidia.com/nim-operator/latest/release-notes.html
NVIDIA: Platform SupportCanonical URL: https://docs.nvidia.com/nim-operator/latest/platform-support.html
NVIDIA: Image Pull SecretsCanonical URL: https://docs.nvidia.com/nim-operator/latest/image-pull-secrets.html
NVIDIA: Caching LLM NIMCanonical URL: https://docs.nvidia.com/nim-operator/latest/cache-llm.html
NVIDIA: Managing NIM ServicesCanonical URL: https://docs.nvidia.com/nim-operator/latest/service.html
NVIDIA: Upgrading NVIDIA NIM OperatorCanonical URL: https://docs.nvidia.com/nim-operator/latest/upgrade.html
NVIDIA: Upgrade NVIDIA NIM MicroservicesCanonical URL: https://docs.nvidia.com/nim-operator/latest/nim-upgrade.html
NVIDIA: NIM Operator DeploymentCanonical URL: https://docs.nvidia.com/nim/large-language-models/2.0.3/deployment/kubernetes-deployment/nim-operator-deployment.html
NVIDIA: Installing the NVIDIA GPU OperatorCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html
The Agent Blast Radius Model: Matching AI Agent Autonomy to Access, Risk, and Reversibility
Role-based access control tells you whether an identity can access a system. That still matters. It is no longer enough. When AI…
The post How to Deploy NVIDIA NIM Microservices on Kubernetes with the NIM Operator appeared first on Digital Thought Disruption.