How Do You Protect a GPU-Backed Workload When Snapshots and Mobility Have Limits?

TL;DR

A GPU-backed workload is not recoverable merely because its virtual disks were copied or its Kubernetes manifests were committed to Git. vGPU, MIG-backed vGPU, PCI passthrough, Enhanced DirectPath, RDMA, and bare-metal GPU configurations expose different snapshot, suspend, migration, and failover boundaries. Some support controlled mobility within a narrow compatibility matrix. Others deliberately trade those virtualization features for direct hardware access.

The durable recovery design should therefore protect the workload in layers:

• Preserve model artifacts, datasets, notebooks, checkpoints, databases, configuration, secrets, and platform state outside the GPU compute instance.
• Treat the VM, pod, or bare-metal node as replaceable compute whenever the application permits it.
• Use application-level checkpoint and restart for long-running training.
• Recreate inference endpoints from approved images, model versions, manifests, and policy-controlled secrets.
• Restore persistent data with the consistency method required by the application, not merely the method offered by the storage platform.
• Maintain a tested recovery-site compatibility contract covering GPU model, MIG configuration, hypervisor, vGPU Manager, guest driver, CUDA, firmware, RDMA, storage, and licensing.
• Measure recovery during exercises. Do not declare the platform recoverable because a backup job completed successfully.

Introduction

Traditional virtual-machine protection encourages a useful but dangerous simplification: take a snapshot, copy the virtual disks, retain the VM configuration, and restore the workload somewhere else when needed. Broadcom’s Virtual Disk API documentation reflects this familiar pattern. A backup application normally asks vSphere for a temporary quiesced snapshot, captures virtual disk and VM configuration data, then removes the snapshot [2].

That model works only when the VM can enter the state required by the backup workflow and when the restored VM can be placed on compatible infrastructure.

GPU-backed AI workloads complicate both assumptions.

A VM using time-sliced vGPU is not the same recovery object as a VM using PCI DirectPath I/O. A MIG-backed vGPU introduces a GPU-instance topology that must exist at the recovery site. An RDMA-enabled workload may use a paravirtual adapter, SR-IOV, Dynamic DirectPath, or Enhanced DirectPath, each with a different feature boundary. A bare-metal training node has no hypervisor snapshot at all. Even when a mobility feature is supported, the target GPU, topology, driver stack, and runtime still have to match the documented support matrix.

The practical recovery question is therefore not:

Can the platform take a snapshot of this workload?

It is:

Which parts of the workload are durable, which parts are replaceable, what hardware-dependent state must be recreated, and what evidence proves that the complete service can recover inside its RTO and RPO?

This article builds that layered model. NVIDIA maintains the current vGPU documentation hub as the release entry point for product guides, support matrices, and release notes [1].

The Recovery Question Is Not “Can I Back Up the VM?”

A backup is a stored copy. Recoverability is the demonstrated ability to restore a useful service.

Those are not the same outcome.

A backup product can report success after copying every virtual disk and still leave the organization unable to recover because the secondary site lacks the required GPU profile. A GitOps repository can contain every Kubernetes manifest and still fail to restore a vector database because the persistent volumes were not protected consistently. A model registry can preserve the final model while a training team loses days of work because optimizer state and dataset position were not included in the checkpoint.

GPU-backed recoverability has four separate questions:

• Can the workload state be captured? This includes application data, file systems, databases, checkpoints, cluster state, and configuration.
• Can the compute envelope be recreated? This includes VM or container runtime, drivers, GPU access mode, device profiles, operators, and scheduling policy.
• Can the workload land on available compatible hardware? This includes GPU model, GPU topology, MIG geometry, firmware, host software, licensing, and network devices.
• Can the application resume correctly? This includes transaction integrity, model loading, training restart, data-index consistency, endpoint health, and client reconnection.

A design that answers only the first question is a backup design. A design that answers all four is a recovery design.

Know Which GPU Attachment Model You Actually Have

NVIDIA distinguishes vGPU, GPU passthrough, and bare-metal deployments. Under vGPU, the NVIDIA Virtual GPU Manager running with the hypervisor creates vGPU devices for guest VMs. In passthrough mode, an entire physical GPU is assigned directly to one VM and bypasses the vGPU Manager. Bare metal places the operating system and workload directly on the physical server [5].

MIG adds another layer. A MIG-capable physical GPU can be partitioned into GPU instances. A MIG-backed vGPU can occupy an entire GPU instance with exclusive access to that instance’s engines, or supported platforms can provide time-sliced vGPUs within a GPU instance [5].

These modes should never be collapsed into one generic label such as “shared GPU.” The recovery controls are materially different.

The operational lesson is simple: inventory the actual attachment mode before selecting a backup method.

A vSphere UI label, a VM hardware description, or the presence of a PCI device is not enough to prove which feature set applies. NVIDIA also publishes a cross-hypervisor vGPU feature knowledge base that can help identify which capability requires deeper product-specific validation [8]. The platform team should record the configured mode, physical device, vGPU profile, MIG state, virtual hardware level, host release, and relevant compatibility evidence.

Why Traditional VM Backup Assumptions Break

The common vSphere image-level backup workflow depends on snapshots and Changed Block Tracking. Broadcom’s API guidance describes a temporary snapshot, normally with guest quiescence enabled and memory capture disabled, followed by virtual-disk extraction and snapshot deletion [2].

PCI DirectPath I/O breaks that workflow by design. Broadcom documents that a VM with a PCI DirectPath I/O device cannot be live migrated, cannot perform snapshot operations, and cannot be suspended or resumed [3]. A separate Broadcom knowledge-base article connects the familiar “operation is not allowed in the current state” snapshot failure to PCI passthrough and advises customers to work with the backup vendor when protection must occur without snapshots [4].

The community discussion supplied with this assignment is useful as an operational signal. The thread shows how the generic error can lead administrators through storage-capacity checks, service restarts, and other troubleshooting before hardware attachment becomes part of the diagnosis [11]. The official knowledge base, not the community thread, should govern the final design.

A powered-off snapshot may succeed in some passthrough situations, but that is a maintenance procedure, not evidence that online image-level backup is supported. Broadcom also documents Changed Block Tracking errors in powered-off passthrough scenarios [4]. Building a daily protection policy around VM shutdown may be acceptable for a low-criticality lab, but it usually creates an outage, extends the backup window, and does not solve application consistency by itself.

vGPU requires a different kind of caution. Current NVIDIA documentation supports vGPU migration on VMware vSphere under defined conditions, but migration support is not a universal statement about snapshot-backed backup. NVIDIA’s migration prerequisites include a running VM, the same physical GPU type at the destination, identical ECC configuration, and identical GPU topology, including NVLink width. Unified memory, debuggers, or profilers can disable migration [5], [6].

Therefore:

• Do not infer backup support from vMotion support.
• Do not infer application consistency from a successful VM snapshot.
• Do not infer failover capability from the existence of another GPU host.
• Do not infer that a restored VM will start merely because its disks were restored.
• Do not assume that all passthrough technologies have the same feature boundary.

Each product-specific statement must be validated against the exact deployed release, device, profile, guest, and backup integration.

Build a Layered Recovery Model

The strongest design does not make the GPU compute instance the only source of truth. It preserves durable state in systems that can be protected, replicated, audited, and restored independently.

The diagram below shows the recovery sequence. Notice that compatible GPU infrastructure appears late in the chain. The organization first restores or validates its durable sources, then recreates the compute envelope, then places the workload on qualified hardware.

This model separates five recovery layers:

• Artifact layer: code, images, models, dependencies, manifests, templates, and policy.
• Data layer: datasets, databases, indexes, notebooks, checkpoints, logs, and persistent application state.
• Platform layer: hypervisor or Kubernetes control plane, operators, storage classes, networking, identity, certificates, and observability.
• Hardware compatibility layer: GPU, MIG, vGPU, NIC, RDMA, firmware, driver, topology, and licensing.
• Service validation layer: application health, data integrity, endpoint behavior, training restart, performance, RPO, and RTO.

No single backup mechanism covers all five.

Separate Persistent State from Replaceable Compute

The first architectural exercise is to classify every workload component as durable state, reconstructable state, or disposable state.

This classification prevents two expensive mistakes.

The first is overprotecting disposable compute. Copying large caches, container layers, and temporary scratch data can lengthen protection windows without improving recovery integrity.

The second is underprotecting small but critical control artifacts. A missing encryption key, image-pull credential, model revision, storage class, or MIG profile can block recovery even when terabytes of data were restored successfully.

Choose the Recovery Pattern by Workload Packaging

GPU attachment mode matters, but workload packaging determines how state is reconstructed.

VM-Based GPU Workloads

A VM-based AI workload often contains too much local state because it was built as an appliance. The operating system, notebook files, model cache, Python environment, credentials, data extracts, and application logs may all live on the same virtual disks.

The better pattern is to split the VM into layers:

• A rebuildable operating-system and runtime disk.
• Separate persistent data disks or network-mounted data services.
• External model and container registries.
• External secrets and identity services.
• Application-native checkpoints or database backups.
• A machine-readable record of the GPU attachment and driver stack.

For time-sliced or MIG-backed vGPU, image-level VM backup may be part of the design only after the exact backup workflow has been tested with the deployed vSphere and NVIDIA release. The test must include snapshot creation, quiescence behavior, backup completion, restore, vGPU reattachment, guest-driver initialization, and application startup. A successful snapshot alone is not enough.

For PCI DirectPath I/O, assume that the normal online snapshot path is unavailable [3], [4]. Use one or more of the following instead:

• Guest-agent file or volume backup.
• Application-native database backup.
• Storage-array snapshots coordinated with application quiescence.
• Replication of persistent data volumes.
• Scheduled checkpoint and restart.
• Powered-off image capture for selected low-change system disks.
• Automated VM recreation from a template with the passthrough device attached at recovery time.

The VM definition should not hard-code a recovery path to one PCI address unless the runbook explicitly handles remapping at the destination.

Containerized GPU Workloads

A containerized workload is usually easier to reconstruct because the application image and desired state can be externalized. That does not make it stateless.

The recovery set normally includes:

• Container image by immutable digest.
• Deployment, StatefulSet, Job, Service, Ingress, Gateway, policy, and configuration manifests.
• GPU resource request, node selectors, affinity, tolerations, runtime class, and scheduling policy.
• Persistent-volume claims and storage classes.
• Database or object-store backups.
• Model and adapter versions.
• Secret references and workload identities.
• Operator versions and custom resources.
• Cluster DNS, certificates, network policy, and service exposure.

A restored container image cannot compensate for a missing model, an incompatible driver, an absent storage class, or an unavailable GPU profile.

Kubernetes, Operators, and GitOps

GitOps restores desired state, not all state.

A Git repository can recreate namespaces, workloads, policies, services, and custom resources. It does not automatically restore etcd, persistent volumes, external databases, model artifacts, secret-manager contents, certificate authorities, load-balancer addresses, or DNS records.

For a Kubernetes-based AI platform, define the recovery order explicitly:

• Restore or rebuild the Kubernetes control plane.
• Restore required cluster identity, certificates, encryption configuration, and external KMS connectivity.
• Install storage, networking, ingress, observability, and GPU platform dependencies.
• Install operator custom-resource definitions and controllers at validated versions.
• Restore secret references and workload identities.
• Restore persistent data and application-native backups.
• Reconcile application custom resources and workloads from Git.
• Confirm GPU discovery and scheduling.
• Run application-level recovery tests.

Kubernetes documents etcd as the backing store for cluster data and recommends a backup plan. Its recovery procedure requires version awareness and coordinated control-plane handling [12]. Kubernetes also notes that API data is stored in etcd without at-rest encryption by default unless an encryption provider is configured [19]. A protected etcd snapshot without the necessary encryption keys may be unusable. An unencrypted snapshot may expose sensitive cluster data.

Use Checkpoint and Restart for Training Workloads

Long-running training should not depend on hypervisor snapshot semantics for progress recovery.

The correct recovery primitive is an application checkpoint written at a business-appropriate interval to durable storage. PyTorch Distributed Checkpoint, for example, supports parallel save and load across ranks and can re-shard at load time across differing cluster topologies within the framework’s supported behavior [16]. That capability is useful, but it does not make every checkpoint universally portable across GPU models, framework releases, or parallelism strategies.

A production training checkpoint should normally include more than model weights:

• Model parameters.
• Optimizer state.
• Learning-rate scheduler state.
• Automatic mixed-precision scaler state where used.
• Random-number-generator state.
• Current epoch, global step, and microbatch position.
• Dataset shard, sampler, or streaming offset.
• Gradient-accumulation state where required.
• Training code commit.
• Configuration and hyperparameters.
• Container image digest.
• Framework, CUDA, library, and driver baseline.
• World size, tensor parallelism, pipeline parallelism, and checkpoint format.
• Checkpoint manifest, checksums, completion marker, and retention metadata.

The checkpoint process must be tested under interruption. A file that exists is not automatically a valid checkpoint. The runbook should select a checkpoint, restore it to an isolated job, resume training, pass a known validation step, and confirm that loss, step count, and data position behave as expected.

Checkpoint frequency is an RPO decision. If the business accepts losing 30 minutes of training progress, a 30-minute checkpoint objective may be reasonable. If checkpoint creation takes 20 minutes and saturates shared storage, the architecture must address asynchronous checkpointing, incremental formats, local staging, or a different RPO.

Recreate Model-Serving Endpoints from Approved Artifacts

Inference services are usually better recovered by reconstruction than by preserving the memory state of a running VM or pod.

For NVIDIA NIM Operator deployments, a NIM service is represented by a NIMService custom resource. The specification identifies the image, authentication secret, model-cache reference, GPU resources, service exposure, and health-probe behavior [14]. NIM model caching is represented separately through NIMCache and persistent storage [15].

That separation creates a clean recovery pattern:

• Restore the Kubernetes platform and NIM Operator at an approved version.
• Restore registry and NGC access through protected secret references.
• Recreate the NIMCache from a pinned source, model revision, engine profile, and storage policy.
• Recreate the NIMService from a pinned image tag or digest and approved configuration.
• Restore ingress, Gateway, TLS, policy, and service-discovery records.
• Wait for the cache, startup, readiness, and liveness conditions.
• Run a known inference request and compare the result envelope, latency, and telemetry with acceptance criteria.

The model cache may be treated as replaceable when it can be regenerated within the RTO. It may need replication or backup when the model is very large, the recovery site is disconnected, or source download time would violate the service objective.

There is also a lifecycle caveat. NVIDIA documents that deleting a NIMCache can delete the operator-created PVC and cached models when PVC creation was managed by the custom resource [15]. GitOps pruning, namespace deletion, or recovery cleanup therefore needs retention controls. A declarative resource can own destructive lifecycle behavior.

The durable source should be the approved model and configuration, not an untracked cache directory on a failed GPU node.

Application-Consistent and Crash-Consistent Protection Are Not Equivalent

A crash-consistent copy represents storage as it might appear after sudden power loss. An application-consistent copy coordinates with the application so transactions, journals, buffers, and related volumes reach a recoverable point.

Kubernetes Volume Group Snapshot reached general availability in Kubernetes 1.36 and can provide crash consistency across all volumes in a group without application quiescence [13]. That is valuable for multi-volume recovery, but it should not be described as application consistency. Applications that require a clean transactional boundary still need quiescence, native backup, or replay logic.

For RAG and vector platforms, consistency often spans several stores:

• Source documents or object storage.
• Ingestion queue or change-feed position.
• Parsed and chunked content.
• Embedding model version.
• Vector index and metadata store.
• Access-control metadata.
• Retrieval configuration and reranker version.

Restoring only the vector volume can produce an index that does not match the source corpus, embedding version, or authorization state. A recovery design should either capture a coordinated application point or preserve enough lineage and replay information to rebuild the derived index.

Milvus provides a useful product-specific example. Its backup tool protects metadata, segments, and data and publishes a version compatibility matrix for backup and restore [17]. The broader lesson is not that every vector platform uses Milvus Backup. It is that native logical backup and version compatibility should be evaluated before relying only on a generic persistent-volume snapshot.

Replicate Supporting Storage, Not Just Compute

GPU capacity receives the attention, but persistent storage usually determines whether the AI service can return.

The recovery design should identify every storage dependency and its role:

• Object storage for datasets, models, checkpoints, and logs.
• File storage for shared training data, notebooks, model caches, and collaborative workspaces.
• Block storage for databases, VM disks, and stateful Kubernetes services.
• Local NVMe for scratch, cache, shuffle, or temporary checkpoints.
• Registry storage for container images and software artifacts.
• Database storage for metadata, catalogs, experiment tracking, feature stores, and vector indexes.

Replication improves availability, but it is not a substitute for backup. Replication can copy corruption, accidental deletion, malicious encryption, and incorrect lifecycle actions. Recovery normally needs some combination of replication, immutable retention, versioning, logical backup, and isolated copies.

Storage recovery must also reproduce access and security behavior:

• Storage class and policy.
• Access mode such as ReadWriteOnce or ReadWriteMany.
• Availability-zone or site topology.
• Encryption keys and KMS access.
• Identity, export, share, bucket, and object permissions.
• Snapshot class and CSI driver.
• Replication relationship and failover direction.
• Retention and immutability settings.
• Quota and capacity headroom.
• Mount path, endpoint, and DNS behavior.

A recovered volume that cannot mount on the GPU nodes is not recovered. A restored object store without the catalog and permissions is not recovered. A vector database with data but an incompatible restore version is not recovered.

Make Recovery-Site GPU Compatibility a Declared Contract

The recovery site should not be described vaguely as “having GPUs.” It should publish a qualified landing-zone contract.

NVIDIA’s current VMware support matrix identifies supported combinations of ESXi releases, GPUs, and guest operating systems [7]. Its vGPU release notes require compatible vGPU Manager and guest-driver releases. An incompatible combination can prevent the vGPU from loading [6]. CUDA provides documented minor-version and forward-compatibility models, but only within defined limits [18].

The recovery baseline should include at least the following records.

A machine-readable declaration makes this easier to govern. The following illustrative record is not a vendor configuration file. It is an architecture control that identifies what must be changed for the real environment and what must be verified during a recovery exercise.

recovery_landing_zone:
  baseline_date: "2026-07-23"
  workload_class: "production-nim-inference"
  site: "secondary-datacenter"

  virtualization:
    platform: "VMware vSphere"
    esxi_release: "approved-release"
    vm_hardware: "approved-level"

  accelerator:
    attachment_mode: "mig-backed-vgpu"
    gpu_model: "approved-model"
    gpu_count: 1
    ecc_state: "enabled"
    mig_profile: "approved-instance-profile"
    vgpu_profile: "approved-vgpu-type"

  nvidia_software:
    vgpu_manager_branch: "approved-branch"
    guest_driver_branch: "compatible-branch"
    cuda_compatibility: "validated-range"
    licensing: "recovery-entitlement-confirmed"

  platform_dependencies:
    kubernetes_release: "approved-release"
    gpu_operator_release: "approved-release"
    nim_operator_release: "approved-release"
    storage_class: "replicated-rwx"
    secret_provider: "enterprise-vault"

  acceptance:
    gpu_discovery: "required"
    model_digest_match: "required"
    readiness_probe: "pass"
    known_inference_test: "pass"
    measured_rto_minutes: 60

The team should replace every placeholder with an approved value, connect it to compatibility evidence, and fail the recovery test when the declared baseline cannot be satisfied. Successful execution means the workload lands on the qualified site, initializes the GPU, accesses persistent data and secrets, and passes service-level validation. Common failures include a missing vGPU profile, driver mismatch, incorrect MIG geometry, unavailable license service, unsupported RDMA device mode, or insufficient GPU memory.

Set RTO and RPO by Workload Type

One RTO and RPO for the entire AI platform is rarely credible. Different workload classes lose different kinds of value.

The following matrix provides illustrative starting points, not contractual objectives. Each enterprise must adjust the targets through business-impact analysis, data classification, workload economics, and available recovery capacity.

Two planning rules matter.

First, RTO must include GPU acquisition and qualification time. A one-hour endpoint RTO is fictional if the recovery site has no reserved accelerator capacity or requires manual firmware and driver changes before the VM can start.

Second, RPO should reflect the authoritative state. Losing a NIM cache may be acceptable if it can be rebuilt quickly. Losing the approved model revision, checkpoint, vector metadata, or encryption key may not be acceptable at all.

Test Recovery Even When GPU Hardware Is Not Available

GPU scarcity is a real constraint. It should change the test design, not eliminate testing.

Use a tiered recovery-validation model.

Control-Plane and Artifact Test Without a GPU

This test proves that the durable sources are usable:

• Restore Git repositories and release metadata.
• Verify container and model digests.
• Restore databases and persistent volumes into an isolated environment.
• Reconcile manifests with GPU workloads scaled to zero or expected to remain Pending.
• Restore operator custom resources without allowing destructive cleanup.
• Validate secret references, KMS access, certificates, DNS, and network policy.
• Open notebooks and verify dependencies that do not require CUDA execution.
• Validate checkpoint manifests, checksums, and metadata.
• Run schema, catalog, object-count, and vector-collection checks.

A Pending GPU pod can be an expected result in this tier. It proves that the scheduler recognized an unsatisfied hardware dependency. It does not prove that the workload can run.

Functional Test on a Surrogate GPU

A lower-tier or alternate GPU can validate selected application behavior when the software stack supports it. This may prove that a model artifact loads, a notebook executes, or a checkpoint format is readable.

It does not certify production performance, memory fit, distributed topology, vGPU migration, MIG placement, RDMA behavior, or driver compatibility on the actual recovery hardware. The runbook must record which acceptance criteria remain open.

Production-Equivalent GPU Recovery Exercise

This is the only tier that can close the hardware-dependent evidence gap. It should validate:

• GPU discovery and health.
• Required vGPU or MIG profile creation.
• Driver and vGPU Manager compatibility.
• CUDA and framework initialization.
• Model loading and GPU-memory fit.
• Training checkpoint restart and forward progress.
• NIM startup, readiness, and known inference request.
• RAG embedding, reranking, and end-to-end query flow.
• RDMA connectivity and distributed workload behavior where used.
• Measured RTO, performance floor, and operational handoff.

Simulated device plugins and mocked GPU resources can test orchestration logic. They cannot validate GPU kernels, memory behavior, driver initialization, hardware health, or performance.

Require Evidence Before Calling the Platform Recoverable

A platform should not be declared recoverable based on architecture diagrams, backup policies, or vendor capability statements. The declaration requires runbook evidence.

The final runbook should answer these questions without improvisation:

• Which backup or checkpoint is selected?
• Which data is authoritative?
• Which data can be rebuilt?
• Which GPU and device modes are required?
• Which versions are allowed?
• Who reserves recovery capacity?
• Who changes DNS, routing, certificates, and service exposure?
• Who validates the model, database, or training job?
• What constitutes a failed recovery test?
• What is the fallback when compatible GPU hardware is unavailable?
• How is failback performed without losing newer recovery-site state?

Recoverability is a current property. It expires when software, firmware, hardware, topology, credentials, operators, storage, or application behavior changes without revalidation.

A Practical Layered Implementation Sequence

Enterprises can build this capability in manageable phases.

Classify the Workloads

Inventory every AI workload by packaging, GPU mode, state location, dependencies, owner, RTO, and RPO. Identify DirectPath, vGPU, MIG, RDMA, and bare-metal exceptions before selecting tools.

Externalize Durable State

Move models, datasets, notebooks, checkpoints, configuration, secrets, and databases out of local GPU-node storage where practical. Define authoritative systems and retention.

Establish Compatibility Baselines

Create qualified source and recovery-site records. Pin supported combinations and capture evidence from NVIDIA, Broadcom, Kubernetes, storage, backup, and application documentation.

Implement Protection by State Type

Use VM backup where supported, guest or application backup where snapshots are unavailable, product-native database backup for stateful platforms, checkpointing for training, GitOps for desired state, and immutable artifact repositories for images and models.

Automate Reconstruction

Create VM templates, bare-metal images, infrastructure-as-code modules, operator installation procedures, GitOps applications, storage mappings, secret integrations, and endpoint validation scripts.

Exercise the Layers

Run frequent no-GPU artifact and data tests, periodic surrogate-hardware tests, and scheduled production-equivalent recovery exercises. Measure rather than estimate.

Govern Change

Revalidate after changes to GPU hardware, firmware, BIOS, ESXi, vCenter, vGPU Manager, guest driver, CUDA, container images, operators, Kubernetes, CSI, RDMA, model profiles, storage, backup products, or security keys.

Common Failure Modes

Several patterns repeatedly create false confidence.

Backup Success Is Treated as Recovery Success

The backup copied disks, but nobody proved that the restored VM could obtain the required GPU profile or initialize the driver.

Mobility Is Treated as Data Protection

A supported vMotion operation reduces maintenance disruption. It does not create an independent recovery copy or protect against corruption, deletion, or site loss.

A Powered-Off Snapshot Becomes the Default Strategy

The method may work for a lab, but it creates regular outages and can still encounter Changed Block Tracking or application-consistency limitations [4].

GitOps Is Treated as a Complete Kubernetes Backup

Git restores desired state. It does not restore etcd, persistent data, external secrets, KMS keys, registries, object stores, databases, or DNS.

Model Cache Is Treated as the Model Registry

A cache improves startup time. It should not be the only copy of an approved model, particularly when operator lifecycle can delete its PVC [15].

Checkpoints Contain Only Weights

The model reloads, but training cannot resume correctly because optimizer, scheduler, scaler, RNG, or dataset-position state is missing.

Recovery Hardware Is Described Only by GPU Model

The GPU matches, but ECC state, topology, MIG geometry, vGPU profile, driver branch, firmware, license, or RDMA device mode does not.

Storage Replication Is Treated as Backup

The secondary copy faithfully receives corruption or deletion from the primary site.

No-GPU Testing Is Presented as Full Certification

Artifact and control-plane testing is valuable, but it cannot prove GPU initialization, model-memory fit, training progress, RDMA, or production performance.

RTO Assumes Idle GPU Capacity

The runbook is technically correct, but the recovery site has no reserved accelerator capacity when the incident occurs.

Conclusion

GPU-backed workloads force enterprises to separate backup convenience from recoverability engineering.

Time-sliced vGPU and MIG-backed vGPU can retain selected virtualization features when the precise hypervisor, GPU, topology, guest, and software conditions are satisfied. PCI DirectPath I/O deliberately removes key snapshot and mobility capabilities. Enhanced DirectPath and RDMA introduce device-specific matrices rather than one universal rule. Bare metal removes the hypervisor protection layer entirely.

The durable strategy is therefore layered. Protect the model, data, notebook, checkpoint, database, configuration, secret, and platform state in systems designed to preserve them. Make VM, pod, and node reconstruction repeatable. Treat caches and scratch as disposable unless their rebuild time threatens RTO. Publish a recovery-site compatibility contract, reserve capacity, and test the complete service on production-equivalent hardware often enough to keep the evidence current.

A platform is not recoverable because a snapshot exists.

It is recoverable when the organization can rebuild the required compute envelope, restore authoritative state, land the workload on compatible GPU infrastructure, resume the application correctly, and prove the result inside the agreed RTO and RPO.

External References

• [1] NVIDIA: NVIDIA Virtual GPU (vGPU) Software
  Canonical URL: https://docs.nvidia.com/vgpu/index.html
• [2] Broadcom Developer: Design and Implementation Overview
  Canonical URL: https://developer.broadcom.com/xapis/virtual-disk-api/latest/vddkBkupVadp.9.2.html
• [3] Broadcom Knowledge Base: Virtual Machine vMotion is not supported on PCI vSphere Direct Path I/O devices at the VM level
  Canonical URL: https://knowledge.broadcom.com/external/article/401437/virtual-machine-vmotion-is-not-supported.html
• [4] Broadcom Knowledge Base: Creating Snapshot of vm using shared GPU fails with the error: Operation Failed! “The Operation is not allowed in the current state”
  Canonical URL: https://knowledge.broadcom.com/external/article/394566/creating-snapshot-of-vm-using-shared-gpu.html
• [5] NVIDIA: Virtual GPU Software User Guide
  Canonical URL: https://docs.nvidia.com/vgpu/latest/grid-vgpu-user-guide/index.html
• [6] NVIDIA: VMware vSphere
  Canonical URL: https://docs.nvidia.com/vgpu/latest/grid-vgpu-release-notes-vmware-vsphere/index.html
• [7] NVIDIA: VMware vSphere ESXi Support
  Canonical URL: https://docs.nvidia.com/vgpu/latest/product-support-matrix/vmware-vsphere.html
• [8] NVIDIA: NVIDIA Virtual GPU (vGPU): Comprehensive Knowledge Base on vGPU Features Across Hypervisors
  Canonical URL: https://docs.nvidia.com/vgpu/knowledge-base/latest/index.html
• [9] VMware Cloud Foundation Blog: Why Enhanced DirectPath Wins for High-Performance Apps
  Canonical URL: https://blogs.vmware.com/cloud-foundation/2026/04/20/why-enhanced-directpath-wins-for-high-performance-apps/
• [10] Broadcom Knowledge Base: Choosing a network adapter for a virtual machine
  Canonical URL: https://knowledge.broadcom.com/external/article/321259/choosing-a-network-adapter-for-a-virtual.html
• [11] Broadcom Community: snapshot failing – Operation not allowed in current state
  Canonical URL: https://community.broadcom.com/vmware-cloud-foundation/discussion/snapshot-failing-operation-not-allowed-in-current-state
• [12] Kubernetes: Operating etcd clusters for Kubernetes
  Canonical URL: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
• [13] Kubernetes: Kubernetes v1.36: Moving Volume Group Snapshots to GA
  Canonical URL: https://kubernetes.io/blog/2026/05/08/kubernetes-v1-36-volume-group-snapshot-ga/
• [14] NVIDIA NIM Operator: Managing NIM Services
  Canonical URL: https://docs.nvidia.com/nim-operator/latest/service.html
• [15] NVIDIA NIM Operator: Caching NIM Models
  Canonical URL: https://docs.nvidia.com/nim-operator/latest/cache.html
• [16] PyTorch: Getting Started with Distributed Checkpoint (DCP)
  Canonical URL: https://docs.pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html
• [17] Milvus: Milvus Backup
  Canonical URL: https://milvus.io/docs/milvus_backup_overview.md
• [18] NVIDIA: CUDA Compatibility
  Canonical URL: https://docs.nvidia.com/deploy/cuda-compatibility/latest/index.html
• [19] Kubernetes: Encrypting Confidential Data at Rest
  Canonical URL: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

The post How Do You Protect a GPU-Backed Workload When Snapshots and Mobility Have Limits? appeared first on Digital Thought Disruption.