TL;DR
No network fabric is universally superior for enterprise AI.
A well-designed Ethernet network is normally the right foundation for out-of-band management, in-band platform services, north-south inference, registry access, and many storage flows. It is also sufficient for single-node training and inference workloads that do not exchange latency-sensitive data across GPU nodes.
A specialized RDMA fabric becomes important when inter-node communication sits on the critical path of useful GPU work. Frequent all-reduce operations, tensor or expert parallelism across nodes, large synchronous jobs, direct storage-to-GPU paths, and expensive GPU idle time all strengthen the case for RoCEv2 or InfiniBand.
RoCEv2 preserves an Ethernet and IP operating model, but it requires disciplined congestion engineering across hosts, NICs, switches, queues, and telemetry. InfiniBand provides a purpose-built RDMA fabric with its own subnet-management and operational model. Either can perform well when the complete stack is validated. Either can fail badly when treated as a simple switch-speed decision.
The practical answer is to classify traffic first, measure the workload’s communication pattern, decide whether RDMA is required, and only then select the fabric that the organization can operate and support.
Introduction
AI infrastructure discussions often jump directly from GPU count to switch speed. The design meeting quickly turns into a debate over 200, 400, or 800 gigabits per second, followed by a second debate over Ethernet, RoCE, or InfiniBand.
That sequence is backwards.
The network requirement is created by the workload’s data movement, synchronization, and failure behavior. An inference platform serving independent requests has a different network profile from a multi-node training job that pauses every step for collective communication. A storage path feeding large sequential datasets has different congestion behavior from an all-to-all mixture-of-experts exchange. A Kubernetes control plane should not share the same failure assumptions as a GPU compute rail.
The architect’s job is therefore not to select the fastest fabric on a product sheet. It is to decide which traffic needs ordinary resilient IP connectivity, which traffic needs high throughput, and which traffic needs an engineered RDMA path with tightly controlled latency and loss.
This article uses NVIDIA’s current enterprise reference architectures and Network Operator 26.4 documentation as the July 2026 support baseline. It also incorporates IEEE Ethernet and data-center-bridging standards, Dell PowerScale storage guidance, and community reports that expose recurring implementation mistakes. Those community discussions are treated as diagnostic signals, not as substitutes for vendor support matrices.
Scope, Assumptions, and Terminology Guardrails
The title presents Ethernet, RoCE, and InfiniBand as three choices, but that shorthand needs correction.
RoCE is Ethernet. RoCEv2 carries RDMA traffic over UDP/IP on an Ethernet fabric. The real comparison is usually among conventional TCP or UDP Ethernet, RDMA over Ethernet, and native InfiniBand RDMA.
This article focuses on data-center and campus-adjacent AI fabrics inside one operational site or availability zone. It does not recommend stretching a lossless fabric across a wide-area multicloud network. Cross-cloud and cross-region AI workflows should normally use standard routed connectivity, data replication, object transfer, checkpoint movement, or application-level distribution. The tightly controlled compute fabric should end where its failure and congestion domain can still be engineered.
The term lossless fabric also needs restraint. It does not mean that links never experience errors or that packets can never be lost. It means that the architecture is engineered to prevent loss for selected traffic classes under the validated congestion envelope. That outcome depends on endpoint congestion control, switch queues, buffer headroom, topology, cabling, firmware, and operating discipline.
Finally, GPU count is a useful sizing input, but it is not the decision by itself. Eight GPUs inside one NVLink domain can create less scale-out traffic than four GPUs split across four servers. A 64-GPU inference farm running independent replicas may be less network-sensitive than a 16-GPU job using cross-node tensor parallelism.
Start With Traffic Planes, Not Fabric Brands
Enterprise AI platforms carry several traffic classes at the same time. They should be designed separately even when some of them share physical switches.
Traffic planeTypical flowsDominant behaviorDesign biasOut-of-band managementBMC, switch management, console, power controlLow bandwidth, high operational importancePhysically separate, simple, redundant, reachable during platform failureIn-band management and platform servicesKubernetes API, image registry, DNS, identity, monitoring, logging, model control planeBursty, many small flows, dependent on stable IP servicesResilient Ethernet with clear security and quality-of-service policyStorage and dataDataset reads, checkpoint writes, model artifacts, vector data, NFS, object storage, backupSustained throughput, bursts, metadata sensitivity, recovery trafficHigh-capacity Ethernet or validated storage fabric, often physically or logically isolatedEast-west GPU computeNCCL collectives, gradients, activations, expert exchange, distributed inference stateSynchronous, latency-sensitive, bursty incast, throughput-sensitiveDedicated or strongly isolated RDMA fabric when communication is on the critical pathNorth-south inferenceAPI requests, token streams, client responses, gateway trafficRequest-response, fan-in and fan-out, tail-latency sensitiveResilient routed Ethernet, load balancing, security, and measured oversubscription
The architecture below shows why a single statement such as “the AI network is 400 gigabit Ethernet” is incomplete.
The most expensive design mistake is allowing a workload burst in one plane to destabilize another. A checkpoint storm should not make the Kubernetes API unreachable. A pause-frame problem on an RDMA priority should not freeze ordinary storage or management traffic. A failed fabric manager should not remove the only path to the switches it manages.
Scale-Up and Scale-Out Communication Change the Answer
AI network design becomes clearer when the architect separates scale-up from scale-out communication.
Scale-Up Communication
Scale-up communication occurs inside a tightly coupled compute domain. Depending on the platform, GPUs may communicate through NVLink and NVSwitch, through PCIe peer-to-peer paths, or through a combination of both. The key characteristic is that communication remains inside the server or rack-scale GPU domain.
When a model, its parallelism strategy, and its memory requirements fit inside that domain, the external compute fabric may not be the performance limiter. This is one reason inference-focused designs can often omit a dedicated GPU east-west fabric.
Scale-Out Communication
Scale-out communication crosses server or rack boundaries through network adapters and switches. At that point, topology, NIC-to-GPU locality, congestion behavior, routing, and collective communication efficiency become critical.
Different parallelism patterns stress the fabric differently:
Data parallel training commonly uses all-reduce operations at step boundaries. The network can determine how long every GPU waits before the next step.
Tensor parallelism exchanges model-state fragments frequently. Cross-node placement can make latency and bandwidth part of nearly every layer execution.
Pipeline parallelism moves activations between stages. Imbalanced stages and network delay create pipeline bubbles.
Mixture-of-experts models can generate heavy all-to-all traffic when tokens are routed to experts on other nodes.
Independent inference replicas generally need north-south capacity but little GPU-to-GPU scale-out traffic.
Disaggregated or model-parallel inference can reintroduce sensitive east-west traffic through remote state, prefill-decode separation, or cross-node tensor exchange.
The first design question should therefore be:
Does the workload cross a node boundary during its performance-critical execution path?
When the answer is no, ordinary high-quality Ethernet may be sufficient. When the answer is yes, the next question is how often the workload synchronizes and how much GPU time is lost while it waits.
What Ethernet, RoCE, and InfiniBand Actually Change
CriterionConventional EthernetRoCEv2 EthernetInfiniBandPrimary transport modelTCP, UDP, HTTP, storage and application protocolsRDMA over UDP/IP on EthernetNative InfiniBand transport and RDMARoutabilityStandard IP routingIP-routable, subject to validated RoCE designRouted within the InfiniBand fabric through its fabric-control modelCongestion modelDrops, queues, TCP control, application retriesECN, endpoint congestion control, selected PFC use, or a validated lossy-RoCE designCredit-based link behavior plus InfiniBand congestion and routing controlsFabric controlFamiliar Ethernet switching and routingEthernet switching plus DCB, RDMA, NIC, queue, and congestion tuningSubnet manager, partitions, fabric routing, and InfiniBand managementKubernetes exposurePrimary CNI and normal interfacesMultus plus RDMA device plugin, SR-IOV, host-device, or shared device modelMultus plus InfiniBand CNI, SR-IOV or shared device model, PKeys, and subnet managementOperational skill profileBroad enterprise networking skill baseAdvanced Ethernet plus RDMA and host-stack expertiseDedicated InfiniBand fabric and subnet-management expertiseStrong fitManagement, north-south inference, registries, general storage, independent replicasOrganizations wanting RDMA with an Ethernet and IP operating modelDedicated large-scale training and HPC environments with mature IB operationsMain trapAssuming link speed alone solves distributed-compute latencyTreating RoCE as ordinary Ethernet with jumbo frames enabledTreating the Kubernetes operator as the fabric control plane
Conventional Ethernet
Conventional Ethernet remains the default enterprise fabric because it is routable, observable, widely supported, and compatible with existing security and operations tooling. It can use 25, 100, 200, 400, or faster links without becoming an RDMA fabric.
That distinction matters. A 400-gigabit TCP path may offer excellent aggregate throughput while still creating more CPU involvement, protocol overhead, tail latency, and synchronization variability than an engineered RDMA path. Conversely, a badly configured RoCE path can perform worse than a stable TCP design.
RoCEv2
RoCEv2 preserves Ethernet cabling, switching, and IP addressing while allowing applications to use RDMA semantics. It is attractive when an organization wants one networking technology family across compute, storage, and services, or when the network team already operates advanced data-center bridging and telemetry.
RoCE is not activated by one switch command. It is an end-to-end behavior that includes:
supported NIC and GPU combinations
host and container drivers
traffic-class mapping
ECN marking and endpoint reaction
PFC scope when PFC is used
buffer thresholds and headroom
MTU consistency
routing and path symmetry
application use of RDMA
operational telemetry under congestion
NVIDIA Spectrum-X is an example of a validated Ethernet platform that coordinates switches, SuperNICs, software, and congestion behavior. It should not be reduced to “generic Ethernet with PFC.”
InfiniBand
InfiniBand is a purpose-built switched fabric with native RDMA semantics and a mature history in high-performance computing. It introduces a separate operational model, including a subnet manager, fabric routing, partitions, link-state and congestion telemetry, and often a dedicated management platform.
That separation can be an advantage. A dedicated compute fabric avoids interaction with ordinary enterprise traffic and can make the performance envelope easier to preserve. It can also add hardware, skills, tooling, and lifecycle boundaries that the organization does not already possess.
InfiniBand is not automatically faster in every enterprise scenario. It is strongest when the workload actually benefits from its fabric characteristics and the operations team can maintain the complete system.
When a Well-Designed Ethernet Network Is Sufficient
Standard Ethernet is normally sufficient when one or more of the following conditions apply:
Training or inference remains inside one server or one scale-up GPU domain.
Inference is delivered through independent replicas with no cross-node model parallelism.
The primary traffic is API, token streaming, image pull, monitoring, logging, registry, or control-plane communication.
Storage performance targets can be met through NFS over TCP, object storage, block storage, or another validated IP storage path.
The platform is still a proof of value and representative testing shows that the network is not limiting useful GPU work.
Virtual-machine mobility, operational simplicity, or multitenant abstraction is more valuable than direct device access.
The team does not yet have the telemetry, automation, and change-control maturity required to operate an RDMA fabric safely.
This does not mean the Ethernet design can be casual. It still needs redundant leaf switches, sufficient uplink capacity, consistent MTU, resilient routing, storage-path diversity, failure-domain analysis, and useful telemetry.
The practical design principle is to avoid paying for complexity before the workload proves that it needs it, while preserving a credible expansion path. Retrofitting dedicated compute NICs, optics, rails, and switches later can require downtime and re-cabling, so an inference-first platform should still reserve rack space, PCIe capacity, switch ports, and cable routes for future training.
When a Specialized RDMA Fabric Becomes Necessary
A specialized RDMA fabric becomes a strong requirement when inter-node communication consumes a material portion of job time or creates unacceptable variability.
The strongest indicators are:
The job uses frequent synchronous collectives across nodes.
Tensor, pipeline, or expert parallelism crosses the server boundary.
A slow rank delays every other rank.
GPU utilization drops during communication phases even though compute and memory capacity are available.
The platform must maintain predictable completion time for expensive long-running jobs.
The cluster spans multiple racks and ordinary oversubscription creates repeatable step-time penalties.
GPUDirect RDMA or GPUDirect Storage is part of the supported workload design.
Checkpoint, data-loader, and compute traffic compete on a shared path.
The cost of idle accelerators exceeds the cost of the specialized fabric and its operations.
There is no responsible universal threshold such as “InfiniBand is required above 32 GPUs.” GPU count is a proxy for communication scale, not the actual requirement. A better threshold is measurable:
Add or upgrade the RDMA compute fabric when representative multi-node jobs cannot meet throughput, completion-time, or GPU-efficiency objectives after software placement and topology issues have been corrected.
GPUDirect RDMA Matters Because It Changes the Data Path
Without a direct peer path, network data may be staged through system memory and involve additional CPU copies or coordination. GPUDirect RDMA allows a supported NIC or DPU to exchange data directly with GPU memory across PCIe. GPUDirect Storage applies a related direct-path model between storage and GPU memory.
The benefit is not only lower latency. Avoiding bounce buffers can reduce CPU overhead and improve effective bandwidth. That can be valuable when many GPUs exchange large tensors or when storage streams data directly into GPU memory.
GPUDirect RDMA is also a support-stack feature, not merely a hardware capability. The architect must validate:
GPU, NIC, DPU, and server compatibility
PCIe root-complex and NUMA placement
IOMMU and Access Control Services behavior
GPU driver, CUDA, kernel, and network-driver alignment
DMA-BUF or legacy peer-memory method
container runtime and operator versions
firmware and switch operating-system versions
Kubernetes resource exposure
application and communication-library use of the direct path
NVIDIA’s current GPU Operator guidance recommends DMA-BUF where the supported kernel and open GPU kernel module allow it. Legacy peer-memory integration remains relevant for other combinations. The selected method must follow the current platform matrix rather than a copied installation command from an older article.
GPUDirect does not remove the fabric bottleneck. It makes the endpoint path more efficient, which can expose congestion, rail imbalance, or switch oversubscription more clearly.
Kubernetes Needs Both a Network Attachment and an RDMA Resource
Kubernetes adds a control and scheduling layer to the physical fabric. NVIDIA Network Operator can manage networking drivers, device plugins, secondary-network components, and policies. NVIDIA GPU Operator manages the GPU software stack. Multus allows a pod to receive a secondary high-performance interface while the primary CNI continues to carry cluster and service traffic.
A pod does not gain working RDMA merely because a secondary interface appears. It normally needs two things:
a Multus network attachment that connects the pod to the intended fabric
an extended resource request that causes the scheduler and device plugin to allocate the RDMA-capable device
The following example is intentionally conceptual. The network name and resource key must match the Network Operator, SR-IOV, or shared-device policy deployed in the target cluster.
apiVersion: v1
kind: Pod
metadata:
name: distributed-trainer
annotations:
k8s.v1.cni.cncf.io/networks: ai-rdma-rail0
spec:
containers:
– name: trainer
image: registry.local/ai/trainer:approved
securityContext:
capabilities:
add:
– IPC_LOCK
resources:
requests:
nvidia.com/gpu: “1”
rdma/rail0: “1”
limits:
nvidia.com/gpu: “1”
rdma/rail0: “1”
Successful scheduling is not sufficient validation. Inside the pod, the operator should confirm the secondary interface, RDMA device, GID or LID state, selected rail, GPU visibility, and actual collective performance.
Shared RDMA Device Versus SR-IOV
Exposure modelStrengthTradeoffBest fitRDMA shared-device pluginSimpler host networking, efficient sharingWeaker device-level isolation and less deterministic allocationTrusted clusters, labs, or controlled shared environmentsSR-IOV virtual functionDedicated VF, stronger isolation, predictable resource accountingFinite VF inventory, more lifecycle complexity, PF/VF and IPAM planningMultitenant clusters and production workloads needing direct allocationHost-device attachmentStraightforward direct interface assignmentDevice becomes unavailable to other workloads and scheduling is rigidSpecialized nodes with static workload placementPCI passthrough to a VMNear-native control in the guestDevice dedication, IOMMU dependence, reduced mobilityPerformance-focused virtualized nodes with fixed placementParavirtualized vNICOperational flexibility and VM mobilityUsually not the direct GPUDirect or native RDMA data pathManagement, application, and conventional storage traffic
Current Network Operator documentation supports specific KubeVirt SR-IOV VFIO combinations, requires IOMMU, and states that live migration is not supported for VMs using that passthrough model. Host-side RDMA is also unavailable for a VF passed through with VFIO because the guest owns the device.
For vSphere and other hypervisors, direct GPU-to-NIC peer paths add further topology and configuration requirements. A VM can have both a flexible management vNIC and a passed-through performance NIC, but the platform team must treat the latter as a placement and recovery constraint.
The Support Matrix Is Part of the Architecture
As of the July 2026 research baseline, NVIDIA Network Operator 26.4 documents exact combinations of operating system, Kubernetes version, distribution, adapter, GPU, and feature support. The matrix includes explicit Rancher RKE2 combinations rather than treating every Kubernetes distribution as equivalent. GPUDirect RDMA currently requires a compatible GPU Operator release and validated networking stack.
The same documentation warns against casually mixing component versions because the operator release is tested as a coordinated set. That warning should become a lifecycle rule:
Pin the operator, driver, CNI, device-plugin, firmware, kernel, and GPU software baseline as one tested platform release.
RoCE Requires Congestion Engineering, Not Just Fast Ethernet
RoCEv2 can provide excellent performance, but its reliability depends on a coherent congestion design.
PFC, ECN, and Endpoint Congestion Control
Priority-based Flow Control, standardized through IEEE 802.1Qbb, can pause one Ethernet priority without pausing every traffic class on the link. It is intended to prevent buffer overflow for selected traffic, not to make the entire network lossless.
Explicit Congestion Notification allows a switch to mark traffic before queues overflow. The receiving endpoint can return congestion feedback, and the sender’s congestion-control algorithm reduces its rate. IEEE 802.1Qau defines Ethernet congestion-notification concepts, while IEEE 802.1Qaz covers Enhanced Transmission Selection and Data Center Bridging Exchange behavior used to coordinate traffic classes.
A healthy RoCE control loop looks like this:
The operational danger is overusing PFC. Pause propagation can create head-of-line blocking, congestion spreading, or a pause storm. PFC should normally be limited to the intended RDMA traffic class, protected with watchdog behavior, and observed through transmit, receive, duration, and queue counters.
NVIDIA documentation also describes lossy RoCE designs that rely on ECN and congestion control without traditional PFC behavior. That option should only be adopted as a validated end-to-end architecture. It is not permission to disable PFC in a design that was tested with it.
Buffer and Headroom Configuration
Switch buffer configuration must reflect port speed, cable distance, MTU, pause-response time, and burst behavior. A copied threshold from a different topology can either waste buffers or allow loss before congestion control reacts.
The network team should understand:
which queue carries RDMA
where ECN marking starts
where PFC pause starts
how much headroom exists per port and priority
whether dynamic buffers can starve another class
how incast behaves at leaf and spine layers
how a failed or slow receiver affects upstream devices
Lossless behavior must be validated under congestion, not inferred from an idle ping test.
MTU Consistency and Jumbo-Frame Failures
Jumbo frames can reduce per-packet overhead, but consistency matters more than selecting a fashionable number. A common design uses a 9000-byte host or workload MTU with a larger switch port MTU that accommodates Ethernet overhead. The exact supported values depend on the platform.
Common failure patterns include:
physical functions set to jumbo MTU while virtual functions remain at 1500
switch ports set correctly while a bond, VLAN, bridge, or overlay remains smaller
the host path is jumbo-capable but the Multus network attachment uses a default MTU
storage and compute interfaces use different assumptions
one leaf-to-spine uplink remains at its default
the VM vNIC, guest interface, and passed-through VF do not agree
encapsulation overhead reduces the effective tenant MTU
Small probes can succeed while large RDMA messages stall or fragment elsewhere in the stack. Validation must originate inside the actual pod or VM and traverse the same VLAN, VF, rail, switch path, and destination used by the workload.
Rail-Optimized Design Preserves GPU and NIC Locality
A rail is a repeated GPU-to-NIC-to-switch path across compute nodes. Rail optimization connects corresponding NIC or GPU positions through the same leaf or plane so collective traffic can remain local to that rail rather than crossing unnecessarily between switch groups.
Rail optimization starts inside the server. The GPU, NIC, PCIe root complex, CPU socket, and NUMA placement should align. Cabling then preserves that relationship through the fabric. Kubernetes resource names and topology labels should expose the same rail identity so the scheduler does not allocate a GPU on one locality domain and an RDMA device on another.
Multi-plane designs add resilience and aggregate capacity. They should be independent enough that a failed switch, optic, cable group, or software plane does not remove all GPU-to-GPU connectivity. Application libraries such as NCCL can then use or fail over among validated paths.
Oversubscription Must Follow Workload Sensitivity
Oversubscription is not inherently bad. It becomes bad when the workload’s simultaneous demand exceeds the fabric’s available bisection bandwidth often enough to violate the service objective.
Traffic planeOversubscription postureReasonOut-of-band managementHigh oversubscription can be acceptable with redundancyTraffic is light, but reachability must survive failuresIn-band managementModerate oversubscription is normally acceptableBursts matter, but sustained line rate is uncommonNorth-south inferenceSize to measured request, response, and model-loading peaksIndependent requests can often share capacity efficientlyStorage and checkpointUse measured read, write, metadata, rebuild, and recovery burstsCheckpoint synchronization can create severe temporary demandSynchronous GPU computePrefer nonblocking or prove the acceptable ratio through representative testsEvery delayed rank can extend step time across the jobAsynchronous batch or loosely coupled jobsSome measured oversubscription may be acceptableWork can tolerate queueing or variable completion time
For a large synchronous training fabric, a 1:1 nonblocking design is a reasonable starting bias. It is not a universal commandment for every AI network. Storage, inference, and management can use different ratios when evidence supports them.
The test is not whether every port can run at line rate independently. The test is whether the simultaneous communication pattern can run without creating unacceptable GPU wait time, tail latency, or recovery risk.
Storage Traffic Needs Its Own Capacity Model
AI storage design is not a secondary detail. GPUs can be idle because the compute fabric is slow, but they can also be idle because datasets, checkpoints, model shards, or metadata arrive too slowly.
PowerScale and similar AI data platforms often connect over high-speed Ethernet even when the GPU compute fabric uses InfiniBand. That is a valid separation of concerns. The storage fabric can use NFS over TCP, NFS over RDMA, multipath client behavior, GPUDirect Storage, or another validated protocol without forcing the compute fabric to use the same transport.
The storage capacity model should include:
aggregate sequential dataset read rate
checkpoint size, frequency, and completion objective
concurrent checkpoint writes from multiple jobs
small-file and metadata operations
data-preparation and shuffle traffic
model and container image distribution
failure, rebuild, and rebalance traffic
backup, replication, and retention workflows
client-to-storage path diversity
per-client and per-storage-node connection behavior
Dell’s PowerScale guidance highlights an important nuance: multipath and multiple connections can improve aggregate streaming behavior across files and nodes, but they do not make every single-file or lock-sensitive workload scale linearly. The application access pattern still matters.
A converged RoCE fabric can carry compute and storage, but those flows must have explicit traffic classes, capacity protection, failure isolation, and observability. A checkpoint burst should not consume the buffers or queues required by a collective operation.
There is also a current software compatibility detail worth recording in the architecture decision. NVIDIA Network Operator 26.4 documents a conflict in one driver-container configuration between host-side NFS RDMA enablement and host-side NVMe over RDMA. This is exactly the kind of version-specific interaction that disappears from high-level diagrams but can block a production design. Storage protocol, host-driver installation method, and operator configuration must be validated together.
Telemetry Must Explain GPU Wait Time
Traditional interface up or down monitoring is insufficient for AI fabrics. The operations model must connect application symptoms to GPU, PCIe, NIC, switch, and storage evidence.
Minimum Telemetry Set
Collect and correlate:
link utilization by direction and rail
ECN marks and congestion notifications
PFC transmit and receive frames
pause duration and watchdog events
queue occupancy, buffer pressure, and drops
CRC, symbol, FEC, and physical-link errors
InfiniBand congestion indicators such as transmit wait
route, path, PKey, GID, and LID state where applicable
NIC firmware and driver health
PCIe width, speed, replay, and topology
NVLink and GPU peer-path health
NCCL collective latency and bandwidth
per-rank completion time
GPU utilization, memory utilization, and stall periods
storage read, write, metadata, queue, and client-path latency
pod, VF, and RDMA resource allocation
Dashboards should align timestamps across these layers. A graph showing falling GPU utilization beside rising PFC duration and collective latency is operationally useful. Three separate dashboards owned by three teams are not.
Symptoms Commonly Mistaken for GPU or Application Problems
SymptomNetwork or fabric cause to investigateGPU utilization becomes sawtoothed during multi-node runsCollective synchronization delay, congestion, rail imbalance, or storage starvationOne rank consistently finishes lateBad optic, FEC errors, NUMA mismatch, PCIe locality, path asymmetry, or overloaded leafNCCL initialization hangsInterface selection, routing, firewall, PKey, subnet-manager, GID, VF, or device-exposure problemSmall tests pass but large transfers failMTU mismatch, buffer threshold, fragmentation, or one smaller segment in the pathTraining works on one node but fails on twoRDMA path, peer-memory integration, ACS/IOMMU, CNI, fabric routing, or firewall issueCheckpoint time increases as cluster use growsShared storage uplink, metadata bottleneck, queue contention, or competing rebuild trafficPod has a secondary interface but no RDMA deviceMissing extended resource request, device-plugin failure, or wrong resource selectorVM sees the VF but cannot use RDMAGuest driver, VFIO ownership, firmware mode, IOMMU, or unsupported combinationInfiniBand link is physically up but traffic does not flowMissing or unhealthy subnet manager, partition issue, or fabric-routing stateInference tail latency spikes during model rolloutRegistry, model-cache, storage, or north-south congestion rather than GPU execution
NVIDIA’s NCCL troubleshooting guidance correctly starts with topology, low-level fabric, bandwidth, latency, and NIC diagnostics before changing communication-library tuning. That order prevents teams from hiding a physical or configuration fault behind environment-variable experimentation.
Reference Topology for Inference-Focused Private AI
This topology is appropriate when production inference uses independent replicas or keeps model parallelism inside each node.
Design characteristics
Use conventional resilient Ethernet for north-south serving, storage, registries, control-plane services, and monitoring.
Size links from request volume, model-loading behavior, token-streaming traffic, and storage demand rather than GPU count alone.
Keep out-of-band management separate.
Reserve a credible path for future compute NICs or RoCE without buying the full training fabric on day one.
Add dedicated RDMA only when distributed inference, remote GPU state, or direct storage paths prove that east-west communication is material.
This is the lowest-complexity architecture and often the best enterprise starting point.
Reference Topology for Moderate Multi-Node Training
This topology fits a small group of GPU servers running distributed fine-tuning or training, often across two to eight nodes. The node range is a planning example, not a fabric threshold.
Design characteristics
Use a dedicated or strongly isolated RDMA compute plane.
Prefer a nonblocking leaf design at this scale unless testing proves an acceptable alternative.
Keep storage and management on resilient Ethernet unless the validated storage design calls for a dedicated RDMA path.
Use two paths or planes when job availability justifies the cost.
Choose RoCE when the organization can operate DCB, ECN, PFC policy, and host-to-switch telemetry consistently.
Choose InfiniBand when a dedicated fabric, subnet-management model, and established IB operational skill are a better fit.
Validate single-node, two-node, and full-cluster collective performance before production acceptance.
Reference Topology for Large Distributed Training
Large synchronous training requires a fabric designed as part of the compute system rather than as shared data-center connectivity.
Design characteristics
Use a rail-optimized, nonblocking leaf-spine or super-spine topology.
Use multiple independent planes when the platform’s availability and performance objectives justify them.
Preserve GPU, NIC, PCIe, rail, and switch locality through cabling and scheduling.
Deploy dedicated fabric management, telemetry, topology validation, and change control.
Keep storage, service, and out-of-band networks distinct from the compute failure domain.
Treat optics, cabling, firmware, and topology files as controlled platform configuration.
Select InfiniBand or a validated RoCE platform from workload evidence, supportability, and operating maturity, not from ideology.
At this scale, the cost of specialized networking should be compared with the cost of thousands of accelerators waiting at synchronization points.
A Practical Fabric Decision Framework
Use the following sequence before selecting a product or bill of materials.
Decision Matrix
ScenarioDefault directionWhyEvidence that can change the answerIndependent inference replicasConventional EthernetTraffic is mainly north-south and storage-orientedCross-node model parallelism, remote state, or measured east-west bottleneckSingle-node training or fine-tuningConventional Ethernet plus strong storageScale-out collectives are absentDataset or storage path requires validated RDMAModerate multi-node trainingBenchmark first, then RoCEv2 or InfiniBandCommunication may or may not dominateNCCL tests and real job traces show acceptable TCP performanceLarge synchronous trainingDedicated RDMA, rail-optimized, near nonblockingCollective delay directly consumes GPU timeA validated alternative meets completion-time and resilience objectivesShared training and inference platformSeparate traffic classes and often separate compute fabricMixed traffic creates competing bursts and operating risksStrict scheduling and measured convergence prove isolation is sufficientVirtualized multitenant platformEthernet for general traffic, selective SR-IOV or passthrough for RDMADirect devices reduce mobility and increase lifecycle constraintsA validated virtualized GPUDirect design meets recovery and support needsStorage-heavy RAG or data engineeringHigh-capacity Ethernet, optionally storage RDMAData movement may dominate without GPU collectivesDirect-to-GPU storage path and measured client throughput justify RDMASmall team with limited fabric operationsConventional Ethernet until skills and telemetry matureComplexity can reduce availability more than it improves speedManaged or validated solution transfers enough operational burden
Workload Size and GPU Count
Larger jobs increase the probability that the fabric matters, but count should remain a screening question. Record nodes, GPUs per node, rails per node, racks, expected concurrent jobs, and failure reserve. Then map those counts to actual communication.
Model Communication Pattern
The communication pattern is the strongest technical criterion. Measure bytes transferred, collective type, synchronization frequency, rank skew, step time, and the proportion of time spent waiting on communication.
Operational Maturity
RoCE requires mature Ethernet automation, queue and buffer governance, telemetry, and host-fabric coordination. InfiniBand requires subnet-manager, partition, topology, routing, and IB telemetry skills. Conventional Ethernet is simpler, but still needs capacity and failure engineering.
The right fabric is the most advanced design the organization can operate reliably, not the most advanced design it can purchase.
Budget and Reversibility
Budget analysis should include switches, optics, cabling, NICs, DPUs, licenses, fabric management, support, lab capacity, spare parts, training, and operational labor. It should also include the cost of accelerator idle time and the disruption of retrofitting a compute fabric later.
Community Signals Reveal Recurring Design Mistakes
The three community discussions supplied with this assignment reinforce several practical lessons.
The RKE2 discussion shows how operator components can remain not ready even when the cluster itself is healthy. The lesson is not that RKE2 is unsupported. The current platform matrix now lists explicit RKE2 combinations. The lesson is that distribution, operating system, container runtime, Node Feature Discovery, labels, driver state, and operator component readiness must be verified as one stack.
The Network Operator and OpenSM discussion exposes a control-plane misunderstanding. Kubernetes can deploy drivers, CNIs, and device plugins, but an InfiniBand fabric still needs a functioning subnet manager and fabric-management design. The cluster operator does not replace the fabric control plane.
The DGX B200 SR-IOV discussion shows a different boundary. A GPU scheduler such as Run:ai is not inherently required to create SR-IOV resources. The foundational requirements are working physical interfaces, correct firmware and BIOS settings, PF and VF configuration, device-plugin discovery, CNI and IPAM design, and accurate resource requests. A scheduler can add policy and fairness later, but it cannot repair a broken hardware exposure path.
Community posts are useful because they show where experienced teams become confused. They should trigger validation questions, not override current product documentation.
Validate the Fabric Before Procurement and Again Before Production
A defensible decision requires evidence from the intended workload, not only a synthetic bandwidth number.
Establish the Baseline
Start with one node. Validate GPU-to-GPU paths, PCIe and NUMA topology, local storage, framework behavior, and expected single-node throughput. This separates fabric problems from application and compute problems.
Add the Smallest Scale-Out Case
Run the same workload across two nodes. Validate interface selection, RDMA device exposure, GPUDirect use, collective bandwidth, collective latency, MTU, route or partition state, and failure behavior.
Test the Communication Pattern, Not Only a Generic Benchmark
Use collective tests to characterize the fabric, but also run representative training, fine-tuning, inference, and checkpoint workflows. Record:
useful samples or tokens per second
step time and communication share
per-rank skew
GPU utilization during collectives
checkpoint completion time
storage read and metadata latency
inference throughput and tail latency
recovery after path or node loss
Create Congestion Deliberately
Run competing jobs, storage reads, checkpoint writes, image pulls, and telemetry export. Confirm that queue policy and fabric isolation behave as designed. A fabric that only works while empty is not production-ready.
Inject Failures
Disable a link, rail, leaf, or storage path within the approved test environment. Confirm application behavior, alternate-path use, telemetry, alerting, and recovery. For InfiniBand, validate subnet-manager high availability and partition behavior. For RoCE, validate congestion response and PFC watchdog behavior.
Define Acceptance Evidence
The production gate should include:
supported component and version matrix
topology and cabling record
MTU validation from workload endpoints
collective and representative-workload results
congestion and failure-test evidence
telemetry dashboard and alert ownership
rollback and recovery procedure
known limitations and exceptions
lifecycle and upgrade sequence
responsible owner for host, fabric, Kubernetes, storage, and application layers
A green switch dashboard is not enough. The platform is ready when the workload meets its service objective and the operations team can explain why.
Conclusion
Enterprise AI does not need one universal network. It needs a set of traffic planes with different performance, resilience, security, and operational requirements.
Conventional Ethernet is the right default for management, north-south inference, platform services, and many storage paths. It is also the sensible starting point when workloads remain inside one GPU node or when evidence shows that scale-out communication is not limiting useful work.
RoCEv2 and InfiniBand become justified when inter-node communication enters the workload’s critical path. RoCE keeps the design inside an Ethernet and IP ecosystem but demands disciplined congestion engineering. InfiniBand creates a purpose-built RDMA fabric but adds a distinct subnet-management and operating model. Neither choice eliminates the need for topology alignment, version control, telemetry, failure testing, and application-level validation.
The final decision should be traceable: workload communication pattern, GPU and node scale, storage behavior, service objective, operational maturity, support matrix, and total cost. When those inputs are visible, the fabric choice stops being a vendor debate and becomes an engineering decision.
External References
NVIDIA: Networking Physical Topologies – NVIDIA HGX AI FactoryCanonical URL: https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory/latest/networking-physical-topologies.html
NVIDIA: Network Logical Architecture – NVIDIA NVL72 AI FactoryCanonical URL: https://docs.nvidia.com/enterprise-reference-architectures/nvl72-ai-factory/latest/network-logical-architecture.html
NVIDIA: NVLink Networking – NVIDIA DGX SystemsCanonical URL: https://docs.nvidia.com/dgx/dgxgb200-user-guide/networking.html
NVIDIA: Platform Support – NVIDIA Network Operator v26.4.0Canonical URL: https://docs.nvidia.com/networking/display/kubernetes2640/platform-support.html
NVIDIA: NVIDIA Network Operator Deployment Guide with KubernetesCanonical URL: https://docs.nvidia.com/networking/display/kubernetes2640/deployment-guide-kubernetes.html
NVIDIA: GPUDirect RDMA and GPUDirect Storage – NVIDIA GPU OperatorCanonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html
NVIDIA: GPUDirect RDMA 13.3 DocumentationCanonical URL: https://docs.nvidia.com/cuda/gpudirect-rdma/
NVIDIA: RDMA over Converged Ethernet – RoCE – Cumulus Linux 5.10Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-510/Layer-1-and-Switch-Ports/Quality-of-Service/RDMA-over-Converged-Ethernet-RoCE/
NVIDIA: Troubleshooting – NCCL 2.30.7 DocumentationCanonical URL: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html
NVIDIA: InfiniBand Cluster Bring-up ProcedureCanonical URL: https://docs.nvidia.com/networking/display/infiniband-cluster-bring-up-procedure.pdf
NVIDIA: Telemetry – UFM Enterprise User ManualCanonical URL: https://networking-docs.nvidia.com/ufmenterpriseum/6202/telemetry
Dell Technologies: PowerScale Multipath Client Driver and AI EnablementCanonical URL: https://infohub.delltechnologies.com/en-us/p/powerscale-multipath-client-driver-and-ai-enablement/
IEEE Standards Association: IEEE 802.3-2022Canonical URL: https://standards.ieee.org/ieee/802.3/10422/
IEEE Standards Association: IEEE 802.1Qbb-2011Canonical URL: https://standards.ieee.org/ieee/802.1Qbb/4361/
IEEE Standards Association: IEEE 802.1Qau-2010Canonical URL: https://standards.ieee.org/ieee/802.1Qau/4051/
IEEE Standards Association: IEEE 802.1Qaz-2011Canonical URL: https://standards.ieee.org/ieee/802.1Qaz/4741/
NVIDIA Developer Forums: Network Operator in RKE2 Cluster for GPUDirect WorkloadsCanonical URL: https://forums.developer.nvidia.com/t/network-operator-in-rke2-cluster-for-gpudirect-workloads/257924
NVIDIA Developer Forums: K8S NVIDIA Network OperatorCanonical URL: https://forums.developer.nvidia.com/t/k8s-nvidia-network-operator/373411
NVIDIA Developer Forums: Clarification on DGX B200 Kubernetes and SR-IOV InfiniBand SetupCanonical URL: https://forums.developer.nvidia.com/t/clarification-on-dgx-b200-kubernetes-sr-iov-infiniband-setup/352856
How to Build an Evaluation Harness for AI Agents Before Production
TL;DR An AI agent should not reach production because its last ten demonstrations looked impressive. It should reach production only after a…
The post Ethernet, RoCE, or InfiniBand? Designing the Network Fabric for Enterprise AI appeared first on Digital Thought Disruption.