How to Install and Configure VMware NSX with an NVIDIA Spectrum Network Fabric

TL;DR

A reliable VMware NSX deployment on NVIDIA networking depends less on clicking through the NSX Manager wizard and more on getting the physical underlay right first. The NVIDIA Spectrum fabric must provide stable Layer 3 reachability between every ESXi and NSX Edge tunnel endpoint, consistent jumbo MTU, predictable uplink behavior, and resilient routing to the Tier-0 gateway.

This guide modernizes NVIDIA’s original NSX-T installation sequence for current NSX 4.x environments. It uses VDS-backed ESXi transport nodes, Cumulus Linux 5.x with NVUE, routed TEP subnets, redundant NVIDIA leaf switches, and numbered BGP adjacencies between the NSX Edge cluster and the physical fabric.

Introduction

Installing NSX on top of an NVIDIA network fabric is not a simple overlay exercise. It is a coordinated design across two control domains.

NSX owns logical segments, Geneve encapsulation, distributed routing, gateway services, and security policy. The NVIDIA Spectrum fabric owns physical link resiliency, routed underlay reachability, MTU consistency, TEP subnet gateways, and external routing adjacency. Neither layer can compensate for a broken design in the other.

The most common deployment failures happen at the boundary between those domains. A host can show as successfully prepared while its TEP cannot reach another rack. An Edge node can be healthy while its BGP session remains idle. A fabric can pass normal ping tests while silently dropping large Geneve packets because one interface still uses a smaller MTU.

This walkthrough focuses on that boundary. It provides a practical deployment order, reference configuration patterns, validation gates, and troubleshooting guidance for building NSX on NVIDIA Spectrum switches running Cumulus Linux.

What You Will Build

By the end of this guide, you will have a deployment pattern that includes:

• A redundant NVIDIA Spectrum leaf-spine underlay
• Cumulus Linux 5.x configured through NVUE
• End-to-end jumbo MTU for NSX overlay traffic
• Routed TEP subnets with redundant VRR gateways
• VDS-backed ESXi host transport nodes
• Separate NSX overlay and VLAN transport zones
• Redundant NSX Edge nodes in an Edge cluster
• A Tier-0 gateway using BGP to peer with NVIDIA leaf switches
• A Tier-1 gateway and test overlay segment
• A repeatable validation and failure-testing sequence

This is an implementation pattern, not a substitute for the Broadcom interoperability matrix, NVIDIA hardware support information, or your approved low-level design.

Version and Scope Guardrails

The original NVIDIA walkthrough was written for ESXi 7.0 Update 3d, vCenter Server 7.0 Update 3d, and NSX-T Data Center 3.2.1. The architecture remains useful, but several implementation details have changed.

For a current deployment, keep these guardrails in place:

• Use the supported NSX release for your vSphere or VMware Cloud Foundation bill of materials. Do not select a version from a standalone tutorial without checking compatibility.
• Use a vSphere Distributed Switch for ESXi transport nodes on NSX 4.x. N-VDS is not supported for ESXi host preparation in NSX 4.x.
• Treat NVIDIA’s Cumulus Linux commands as version-aware examples. The current NVIDIA deployment guide uses Cumulus Linux 5.x and NVUE, but exact syntax and defaults should be checked against the installed release.
• Validate NIC firmware, driver, switch software, optics, and transceivers as one support chain. Link-up status alone does not establish a supported design.
• Separate pure NSX overlay requirements from EVPN requirements. A routed IP fabric can carry NSX Geneve traffic without deploying VXLAN EVPN in the physical fabric. Add EVPN only when the physical design has an independent requirement for it.

The article uses the term NSX-T where it improves searchability or describes the historical product lineage. Current product documentation generally uses VMware NSX or VCF Networking depending on the platform release.

Reference Architecture

The physical fabric should remain simple from the NSX perspective. It does not need to learn every workload MAC address or tenant segment. Its primary job is to route packets between TEP IP addresses and provide reliable external reachability for the NSX Edge cluster.

The important distinction is that the NVIDIA fabric routes the outer IP header between TEPs. NSX owns the Geneve encapsulation and the logical topology carried inside it.

For north-south traffic, the NSX Edge service router terminates the logical routing path and exchanges routes with the NVIDIA fabric through numbered BGP adjacencies. That routing boundary should be explicit, filtered, monitored, and tested independently from host TEP reachability.

Prerequisites

Platform and Access Requirements

Information to Record Before Deployment

Do not begin the build with only a list of VLAN IDs. Capture the complete relationship between the physical and virtual networks.

The values above are illustrative. A production low-level design should also define route filters, BFD policy, default-route ownership, failure domains, oversubscription, Edge placement, and IP exhaustion thresholds.

Make the Physical Uplink Decision First

NVIDIA documents two useful host attachment patterns:

• Active-active LACP through an MLAG bond
• Non-LAG uplinks using active-standby or another independent-uplink policy

Both can work. The wrong choice is configuring the physical switches for one model and the VDS or NSX uplink profile for the other.

Active-Active LACP Through MLAG

Use this pattern when you need both links forwarding, the VDS LAG is part of the approved design, and the network team is comfortable operating MLAG and LACP as one failure domain.

The following example creates an MLAG bond for one ESXi host on an NVIDIA leaf. Change the physical port, bond name, and MLAG ID to match the actual cabling plan.

nv set interface esxi01 bond member swp1
nv set interface esxi01 type bond
nv set interface esxi01 bond mode lacp
nv set interface esxi01 bond mlag id 1
nv set interface esxi01 bridge domain br_default
nv set interface esxi01 bridge domain br_default stp bpdu-guard on
nv set interface esxi01 bridge domain br_default stp admin-edge on
nv set interface esxi01 bond lacp-bypass on
nv config apply -y
nv config save

A successful result should show the bond up on both MLAG peers with no conflict or protocol-down reason. The VDS must use a matching LAG configuration, and the NSX transport node profile must map its uplinks to the correct VDS LAG.

Non-LAG Active-Standby Uplinks

Use this pattern when operational simplicity and deterministic failover matter more than using both links simultaneously. Each ESXi-facing switch port remains an independent trunk port.

nv set interface swp1 bridge domain br_default
nv config apply -y
nv config save

Repeat the configuration on the peer leaf for the second physical NIC. In NSX, the uplink profile then defines which uplink is active and which is standby.

Do not create an MLAG host bond on the switches for this design. An active-standby NSX or VDS policy connected to an MLAG LACP bond is not a harmless mismatch. It creates inconsistent forwarding and difficult failure behavior.

Configure the NVIDIA Underlay

Validate Physical Connectivity

Use LLDP before configuring logical constructs. It is much faster to correct a crossed cable or unexpected switch port now than after NSX host preparation.

net show lldp
nv show interface

Confirm:

• Every ESXi NIC appears on the expected leaf and switch port
• Leaf-to-spine links are correct
• MLAG peer links are connected as designed
• Interface speed, breakout mode, and MTU are consistent
• No unexpected access-port or native-VLAN configuration exists on host trunks

Set and Verify Jumbo MTU

NVIDIA’s current NSX-T fabric guide recommends jumbo MTU end to end and notes that Cumulus Linux data interfaces default to 9216 bytes. NSX Geneve traffic requires additional encapsulation headroom, so a normal 1500-byte path is not enough for full-size workload frames.

Use a consistent design such as:

• Physical switch data interfaces: 9216
• VDS: 9000
• NSX host switch and uplink profile: 9000
• TEP VMkernel interfaces: inherited from the NSX transport configuration
• Edge data interfaces: aligned to the NSX design

Verify the switches rather than assuming the default is unchanged:

nv show interface

MTU must be correct on every hop. One smaller inter-switch link can produce intermittent packet loss that only appears under larger frames or specific application traffic.

Configure MLAG and VRR

MLAG provides the paired-leaf system needed for dual-attached hosts. VRR provides a shared active-active gateway address for routed TEP subnets.

The following example creates the TEP VLAN and a VRR gateway on the first leaf in a pair:

nv set bridge domain br_default vlan 110
nv set interface vlan110 ip address 10.110.0.252/24
nv set interface vlan110 ip vrr address 10.110.0.254/24
nv set interface vlan110 ip vrr mac-address 00:00:00:00:01:10
nv set interface vlan110 ip vrr state up
nv config apply -y
nv config save

On the peer leaf, use a unique physical SVI address such as 10.110.0.253/24, but configure the same VRR gateway address and VRR MAC.

Verify both MLAG and VRR before moving forward:

nv show mlag
nv show interface
net show clag

The peer must be alive, the peer link must be healthy, and the SVI plus VRR interfaces must show up.

Configure the Routed BGP Underlay

NVIDIA’s reference design uses eBGP, Auto BGP, and BGP unnumbered between leaf and spine interfaces. The pattern keeps the underlay simple while providing equal-cost paths between racks.

A leaf example looks like this:

nv set router bgp enable on
nv set vrf default router bgp autonomous-system leaf
nv set vrf default router bgp neighbor peerlink.4094 remote-as external
nv set vrf default router bgp neighbor swp51 remote-as external
nv set vrf default router bgp neighbor swp52 remote-as external
nv set vrf default router bgp router-id 10.10.10.1
nv config apply -y
nv config save

Use explicit ASNs instead of Auto BGP when that better fits enterprise routing policy, logging, troubleshooting, or route-governance requirements. The architectural requirement is not Auto BGP itself. The requirement is stable routed reachability between every TEP subnet with predictable equal-cost paths.

Advertise the TEP subnets into the underlay and verify that each rack learns the remote TEP routes.

net show bgp summary

Do not proceed until all expected fabric peers are established and the remote TEP prefixes are present in the routing table.

Deploy the NSX Manager Cluster

Deploy the First NSX Manager

Deploy the first NSX Manager appliance from the supported installation media. Record the following values before starting the wizard:

• Management IP address and prefix
• Default gateway
• DNS servers and search domains
• NTP servers
• Hostname and forward/reverse DNS records
• Admin, root, and audit credentials
• Appliance size based on the supported scale profile
• Target cluster, datastore, and management port group

After deployment, use the appliance console to verify the management interface and service state:

get interface eth0
get services

A few optional services may remain stopped by default. The important result is that the management interface is correct, core services are running, DNS resolves, NTP is synchronized, and the UI is reachable through the intended management path.

Register vCenter as the Compute Manager

In NSX Manager, add vCenter as a compute manager under the Fabric configuration. Validate the certificate thumbprint and wait until both registration and connection status are healthy.

Do not treat compute-manager registration as a formality. NSX uses that relationship for cluster discovery, VDS selection, host preparation, Edge placement, and inventory synchronization.

Add the Remaining Manager Nodes

For production, deploy two additional Manager nodes from the NSX UI and form a three-node cluster. Keep the appliance sizes consistent and place the nodes across appropriate host and storage failure domains.

After all nodes are healthy:

• Assign the cluster virtual IP if the design uses one
• Configure backups to an external repository
• Validate name resolution for all nodes and the virtual IP
• Confirm time synchronization
• Test administrative access through the supported management endpoint

Take the first NSX backup before preparing any hosts.

Build the NSX Transport Configuration

Create the Transport Zones

Create at least two transport zones:

• Overlay transport zone: Used by ESXi host transport nodes and NSX Edge nodes for Geneve-backed segments
• VLAN transport zone: Used by NSX Edge uplinks and other VLAN-backed segments that connect to the physical network

Use clear names that describe site, rack, and purpose. Avoid names that only make sense during the installation project.

Create the Host Uplink Profile

The host uplink profile must match the physical attachment decision.

For a non-LAG active-standby design, define:

• Teaming policy: Failover order
• Active uplink: uplink-1
• Standby uplink: uplink-2
• Transport VLAN: Host TEP VLAN for that rack or cluster
• MTU: 9000, when validated end to end

For an LACP design, align the NSX uplink mapping with the VDS LAG and the NVIDIA MLAG bond. Do not copy a legacy N-VDS LAG example directly into NSX 4.x host preparation.

Create the Edge Uplink Profiles

Create Edge profiles separately from the host profile. Edge nodes have different interfaces, traffic roles, and failure behavior.

A typical Edge design needs:

• An overlay or TEP uplink profile
• VLAN uplink mappings for Tier-0 external interfaces
• Named teaming policies when specific VLAN-backed segments must use specific Edge interfaces
• MTU aligned to the physical path

Do not assume the Edge can reuse the host LACP design. The NVIDIA reference design treats Edge external paths as separate numbered links to the leaf switches.

Create TEP IP Pools

Create TEP pools that match the routed fabric design. Per-rack host TEP pools provide clear fault isolation and make route troubleshooting easier.

For each pool, define:

• IP range
• Prefix length
• Default gateway, normally the VRR address on the NVIDIA leaf pair
• DNS only if the deployment workflow requires it

Use a separate Edge TEP pool when the Edge nodes use a different VLAN, subnet, or lifecycle boundary.

Create the Transport Node Profiles

Create a transport node profile for each cluster configuration that differs by VDS, uplink mapping, TEP VLAN, or IP pool.

For NSX 4.x ESXi hosts, select the VDS as the host switch. Then map:

• Overlay transport zone
• Host uplink profile
• TEP IP assignment method
• Correct rack-specific TEP pool
• NSX uplinks to the intended VDS uplinks or LAG

Applying one universal transport node profile to clusters with different rack-local TEP pools is a common design mistake. The profile should express the real physical attachment and IP plan.

Prepare the ESXi Clusters

Apply the transport node profile to a pilot cluster first. Watch both installation status and transport-node health.

A successful result should show:

• NSX installation status: Success
• Configuration status: Success
• Manager connectivity: Up
• Controller connectivity: Up
• VDS status: Healthy
• TEP addresses allocated from the expected pool
• Physical NIC mappings aligned to the low-level design

Do not prepare every cluster in one change window until the pilot host pair has passed MTU, tunnel, failover, and workload tests.

Deploy the NSX Edge Cluster

Deploy at least two NSX Edge nodes for production north-south routing. Place them on different ESXi hosts and, where possible, different rack or failure domains.

For each Edge node, configure:

• Management hostname, IP address, gateway, DNS, and NTP
• Appliance size based on throughput and service requirements
• Management port group
• Overlay transport zone and Edge TEP pool
• VLAN transport zone
• Uplink profile and interface mappings
• Placement that avoids both Edges failing with one host

After deployment, create an Edge cluster and add both nodes.

Validate from the UI and, when needed, the Edge CLI:

get interfaces
get managers
get controllers

The Edge nodes must have healthy management and control-plane connectivity before Tier-0 routing is configured.

Configure Tier-0 Routing to the NVIDIA Fabric

Create VLAN-Backed Uplink Segments

Create one VLAN-backed segment for each physical Tier-0 uplink network. Associate them with the VLAN transport zone and the correct VLAN IDs.

Keep these segments dedicated to routing adjacency. Do not mix random appliance traffic, management traffic, and Tier-0 peering on the same VLAN simply because the subnet has available addresses.

Create the Tier-0 Gateway

Create the Tier-0 gateway and attach it to the Edge cluster.

Choose the HA mode based on services:

• Active-active: Best fit for routing-focused designs that need ECMP and scale-out north-south forwarding
• Active-standby: Required when the Tier-0 must host stateful services that cannot operate in active-active mode

Create external interfaces for the NVIDIA leaf adjacencies. Each interface needs an IP address on its VLAN-backed uplink segment.

Configure Numbered BGP Peering

The NVIDIA design uses numbered BGP peering between NSX Edge uplinks and the leaf switches. Do not assume the unnumbered BGP pattern used inside the leaf-spine fabric also applies to the NSX Edge boundary.

An NVIDIA leaf-side example is:

nv set vrf default router bgp neighbor 172.20.10.2 remote-as 65050
nv config apply -y
nv config save

In NSX, configure:

• Tier-0 local ASN
• BGP neighbor IP for each NVIDIA leaf
• Remote ASN
• Source interface
• BFD only after base peering is stable and both platforms use compatible timers
• Inbound and outbound route filters
• Redistribution rules for the exact connected and static prefixes that should leave NSX

Avoid unrestricted redistribution. A Tier-0 should not advertise every connected object simply because the BGP session is up.

Create the Tier-1 Gateway and Overlay Segment

Create a Tier-1 gateway, connect it to the Tier-0, and create an overlay segment for the test workload.

Attach test virtual machines only after:

• Tier-0 BGP peers are established
• Expected northbound and southbound routes are present
• Host and Edge TEPs have end-to-end reachability
• NSX transport nodes are healthy

Understand the Packet Flow

The deployment is easier to troubleshoot when the team can describe what changes at each hop.

For east-west traffic, the NVIDIA fabric sees routed traffic between TEP IP addresses. It does not need the tenant segment configured as a physical VLAN.

For north-south traffic, the Edge removes the NSX overlay context and forwards traffic through the Tier-0 external interface. The NVIDIA leaf then routes the packet based on the prefixes exchanged through BGP.

Validate the Deployment

Validate the NVIDIA Fabric

Run the following checks on each leaf and spine:

net show lldp
nv show interface
nv show mlag
net show clag
net show bgp summary

Confirm:

• All physical links are up at the expected speed
• MLAG peers are alive and conflict-free
• VRR interfaces are up
• Leaf-spine BGP peers are established
• Remote TEP prefixes are installed
• Edge BGP peers are established on the border leaves

Validate TEP MTU from ESXi

Identify the NSX TEP VMkernel interface and test a remote TEP with the do-not-fragment flag. Replace the VMkernel interface and destination IP with values from the environment.

esxcli network ip interface list
vmkping ++netstack=vxlan -I vmk10 -d -s 8972 10.120.0.11

A successful 8972-byte payload validates a 9000-byte IP path. Test between racks, not only between hosts on the same leaf pair.

Validate NSX Transport Health

In NSX Manager, confirm:

• Host transport nodes are Up and Success
• Edge transport nodes are Up
• Tunnel status is healthy
• Manager and controller connectivity are Up
• TEP IPs match the intended pools
• No VDS uplink or LAG mappings are degraded

Use NSX Traceflow to validate the logical forwarding path before assuming the issue is physical.

Validate Routing and Failure Behavior

Test more than a single successful ping.

Capture routing tables, BGP summaries, interface state, NSX alarms, and packet-loss measurements before and during each failure test.

Troubleshooting Common Failures

Hosts Prepare Successfully but Tunnels Stay Down

Likely causes include:

• Wrong transport VLAN in the host uplink profile
• TEP pool gateway does not match the NVIDIA VRR address
• Remote TEP subnet is missing from BGP
• VDS uplink mapping points to the wrong physical NIC
• One inter-switch link has a smaller MTU
• Host-facing port is configured as access instead of trunk

Start with the TEP IP, VLAN, route, and MTU. Reinstalling NSX software rarely fixes an underlay reachability problem.

BGP Remains Idle or Active

Check:

• Tier-0 interface and leaf SVI or subinterface addresses
• Prefix length on both sides
• Local and remote ASN values
• Correct source interface in NSX
• VLAN allowed on the Edge uplink path
• ACL or control-plane policy blocking TCP port 179
• Duplicate IP address or adjacency configured against the wrong Edge node

Establish basic IP reachability before changing BGP timers or adding BFD.

Only One Host Link Carries Traffic

This may be correct for an active-standby design. Confirm the NSX uplink profile and VDS teaming policy before treating it as a fault.

For LACP, confirm:

• The VDS LAG is up
• Both NVIDIA MLAG peers use the same bond and MLAG ID
• LACP mode matches
• The MLAG system MAC is consistent
• No interface is suspended because of a mismatch

Traffic Fails During Leaf Maintenance

Inspect MLAG peer-link health, VRR state, BGP convergence, and the host attachment model. A dual-cabled server is not automatically resilient if the switch pair does not agree on VLANs, bond state, or gateway ownership.

East-West Works but North-South Fails

That result usually means the overlay is healthy and the problem is at the Edge boundary.

Check:

• Edge TEP health
• Tier-0 external interfaces
• BGP neighbor state
• Route advertisement and filtering
• Default-route direction
• Return routes in the physical fabric
• Edge placement and uplink segment mappings

Small Pings Work but Applications Fail

Suspect MTU first. Validate a large do-not-fragment packet between actual TEPs across the complete path. Also check that the physical switch MTU, VDS MTU, NSX profile MTU, and Edge data-path MTU are aligned.

Operational Handover and Rollback

A successful installation is not complete until the network and virtualization teams can operate it together.

Record:

• Physical port-to-NIC mappings
• MLAG bond IDs and peer-link details
• VRR addresses and MACs
• TEP pools and rack ownership
• VDS uplink or LAG mappings
• Edge interface-to-VLAN mappings
• Tier-0 BGP peers, ASNs, filters, and advertised prefixes
• Expected failure behavior
• Monitoring dashboards and alert owners
• Backup locations and restore procedures

Use a staged rollback sequence. Do not unprepare hosts while workloads still depend on NSX segments.

A safe rollback plan normally reverses the deployment in dependency order:

1. Move or disconnect test workloads from NSX segments.
2. Remove Tier-1 attachments and routing advertisements.
3. Remove Tier-0 BGP neighbors and external interfaces.
4. Remove Edge cluster dependencies before deleting Edge nodes.
5. Unprepare only the pilot cluster after confirming no NSX-backed workloads remain.
6. Revert NVIDIA switch changes from a known saved configuration.

Take NSX and switch backups before each major phase. A rollback should restore a known state, not create a third partially configured state.

Conclusion

Installing VMware NSX with an NVIDIA Spectrum network fabric is a cross-platform architecture exercise disguised as an installation wizard. The Manager appliances, transport zones, TEP pools, and Edge cluster are important, but they only work when the underlay provides deterministic reachability, MTU consistency, resilient gateway behavior, and clean routing boundaries.

The most reliable deployment order is to validate the NVIDIA fabric first, prepare a small NSX pilot second, prove TEP and Edge routing third, and only then scale host preparation across the environment. That sequence exposes physical and virtual mismatches while the blast radius is still small.

Keep the design boundaries explicit. Use VDS-backed host transport nodes on NSX 4.x, route TEP subnets through the NVIDIA fabric, keep the host and Edge uplink models separate, and use numbered BGP adjacencies at the Tier-0 boundary. When those decisions are documented and tested, NSX and Cumulus Linux form a clean, scalable network stack rather than two independent systems that happen to share cables.

External References

• NVIDIA: Cumulus Linux Deployment Guide for VMware NSX-T
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/guides/nsxt/
• NVIDIA: HowTo Install and Configure an NSX-T with NVIDIA Network Fabric
  Canonical URL: https://docs.nvidia.com/networking/display/public/sol/howto%2Binstall%2Band%2Bconfigure%2Ban%2Bnsx-t%2Bwith%2Bnvidia%2Bnetwork%2Bfabric
• Broadcom TechDocs: VMware NSX 4.2.4.1 Release Notes
  Canonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/release-notes/vmware-nsx-4241-release-notes.html
• Broadcom TechDocs: Deploy NSX Manager Nodes to Form a Cluster from the UI
  Canonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/installation-guide/installing-nsx-manager-cluster-on-vsphere/install-nsx-manager-and-available-appliances/deploy-nsx-manager-nodes-to-form-a-cluster-using-ui.html
• Broadcom TechDocs: Create an IP Pool for Tunnel Endpoint IP Addresses
  Canonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/installation-guide/transport-zones-and-transport-nodes/create-an-ip-pool-for-tunnel-endpoint-ip-addresses.html
• Broadcom TechDocs: Adding a Transport Node Profile
  Canonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/installation-guide/transport-zones-and-transport-nodes/add-a-transport-node-profile.html
• Broadcom TechDocs: Configure BGP
  Canonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/administration-guide/tier-0-gateways/configure-bgp-in-nsx.html

The post How to Install and Configure VMware NSX with an NVIDIA Spectrum Network Fabric appeared first on Digital Thought Disruption.