TL;DR
An HCX 9.1 migration should not begin with the Migrate Virtual Machines button. It should begin with a dependency-ordered runbook that proves routing, firewall policy, IP pools, site pairing, Network Profiles, Compute Profiles, Service Mesh health, and Network Extension behavior before a production workload is touched.
The practical sequence is:
Confirm supported versions, backups, DNS, NTP, certificates, routes, and firewall ownership.
Reserve unique, non-overlapping HCX appliance IP pools at both sites.
Establish and validate HCX site pairing over TCP 443.
Create Network Profiles for the required traffic types.
Create Compute Profiles that constrain placement and enable only the required services.
Deploy the Service Mesh and run diagnostics until every required appliance and tunnel is healthy.
Extend one low-risk test network and validate security, MTU, routing, and east-west behavior.
Move a disposable test VM, then a controlled Bulk Migration canary.
Keep the source authoritative until application, network, monitoring, and backup tests pass.
Roll back according to the actual migration state, not by improvising after cutover.
Introduction
HCX failures often appear late, even when their causes were introduced much earlier. A migration may fail at switchover because TCP 8000 is blocked. A Service Mesh may fail because two Network Profiles contain overlapping address ranges. A migrated VM may power on successfully but lose same-subnet connectivity because a strict NSX SpoofGuard profile rejects proxied return traffic.
The team then spends the maintenance window troubleshooting the final symptom instead of correcting the original design dependency.
That is why HCX configuration needs to be treated as a chain of evidence. Every stage should produce a clear pass condition before the next stage begins. Site pairing proves the managers can communicate. Network Profiles prove that the appliance interfaces have valid addressing and reachability. Compute Profiles prove placement, service scope, and resource intent. The Service Mesh proves that the data-plane appliances and tunnels can operate. Network Extension proves the Layer 2 path. The first VM move proves the complete migration workflow.
This runbook is written for architects, VMware administrators, network engineers, application owners, and change managers preparing an HCX 9.1 migration between vSphere-based sites. It focuses on the first controlled implementation path, not every HCX topology or migration service.
Objective and Exit Criteria
The objective is to build a working HCX 9.1 migration path between two vSphere-based sites and complete the first controlled VM move without making the destination authoritative until validation is complete.
The runbook is successful when all of the following are true:
Both HCX Managers report healthy local platform connections.
Site pairing is healthy and remote inventory is visible.
Network Profile address pools are unique, reserved, reachable, and large enough for the selected design.
Compute Profiles contain the intended clusters, datastores, networks, services, and resource reservations.
Service Mesh appliances deploy successfully and report a healthy state.
HCX diagnostics report no unexplained blocked paths.
The selected test network extends successfully.
A test VM can communicate across the extended network.
The first canary migration completes and passes technical and application validation.
The source recovery path remains documented and executable until acceptance.
Scope and Assumptions
This runbook assumes:
HCX 9.1 Managers are deployed or ready to be deployed at both sites.
Both sites are vSphere-based and are within the current HCX interoperability and support boundaries.
The required HCX entitlement is available for the services being enabled.
DNS, NTP, certificate trust, vCenter connectivity, and administrative access are available.
The network team can validate firewall flows from the actual HCX appliance and VMkernel networks, not only from an administrator workstation.
The first production-like test will use a low-risk VM with a defined owner and validation plan.
Network Extension will initially be limited to one test VLAN or NSX segment.
This runbook does not cover OS Assisted Migration, disaster recovery orchestration, cloud-provider-specific restrictions, Mobility Optimized Networking design, or a full subnet gateway cutover.
The HCX Dependency Order
The most important design principle is that HCX is built in dependency order. A green object at one layer does not prove the next layer is healthy, but a failure at an early layer can invalidate everything above it.
The diagram below shows the sequence that should control the change plan.
The operational lesson is straightforward: do not troubleshoot a migration workflow until the Service Mesh is healthy, and do not troubleshoot the Service Mesh until its profiles, routes, pools, permissions, and firewall paths are proven.
Pre-Change Safety Checks
Confirm the Version and Support Baseline
Record the exact HCX Manager and fleet-appliance versions at both sites. Confirm compatibility with the local vCenter, ESXi, NSX, and any cloud-provider endpoint.
Review current HCX 9.1 release information and known issues before the maintenance window. A generic HCX procedure may not account for a version-specific behavior.
Do not assume that two sites reporting “9.1” are on the same patch level. Record:
Complete HCX Manager version and build
IX and NE appliance versions
vCenter and ESXi versions
NSX version when applicable
Approved target baseline
Research and validation date
Back Up the HCX Managers
Configure and verify an HCX Manager backup before modifying site pairing, profiles, or Service Mesh objects.
The change record should include:
Last successful backup timestamp
Backup destination and retention
Restore credentials and ownership
Exact HCX build represented by the backup
Recovery decision owner
Confirmation that the backup is recent enough for the current HCX inventory
A VM snapshot is not a substitute for the supported HCX Manager backup workflow.
Freeze Conflicting Operations
Before deploying or changing the Service Mesh:
Pause vCenter, ESXi, NSX, and HCX lifecycle changes.
Pause backup jobs that create snapshots on the canary VM.
Avoid datastore maintenance affecting HCX appliances.
Avoid DRS or automation rules that repeatedly move HCX appliances.
Confirm there are no existing migrations in a switchover state.
Confirm no Network Extension changes are in progress.
Confirm no Service Mesh resynchronization or redeployment is active.
Assign Operational Ownership
Area
Required owner decision
HCX
Profiles, Service Mesh, diagnostics, migration state
vSphere
Cluster, datastore, folder, permissions, VM state
Network
Routing, NAT, firewall, MTU, packet captures
NSX
Segment security, SpoofGuard, MAC learning, gateway behavior
Application
Functional testing and acceptance
Change management
Go, hold, rollback, escalation
Do not enter the first migration window without named owners who can make decisions. A troubleshooting bridge full of observers but no decision authority will extend the outage.
HCX 9.1 Firewall Ports and Traffic Paths
The official HCX 9.1 port matrix remains the governing source. The table below is a runbook shortlist for the paths most likely to affect site pairing, Service Mesh deployment, Network Extension, Bulk Migration, and Replication Assisted vMotion.
It is not a replacement for the complete Broadcom port and protocol requirements.
Source
Destination
Protocol and port
Operational purpose
Administrator workstation
HCX Manager appliance interface
TCP 9443
Appliance administration, backup, activation, and health review
Initiating HCX Manager
Remote HCX Manager
TCP 443
Site pairing, control-plane communication, and inventory access
HCX Manager
Local vCenter and required ESXi endpoints
TCP 443
API calls, inventory, placement, and workflow control
Source IX uplink
Destination IX uplink
UDP 4500
Encrypted HCX Interconnect data-plane tunnel
Source NE uplink
Destination NE uplink
UDP 4500
Encrypted Network Extension data-plane tunnel
Source ESXi management or provisioning VMkernel
Local source IX management interface
TCP 902
NFC replication path used by replication-assisted workflows
Destination IX management interface
Destination ESXi management or provisioning VMkernel
TCP 902
Replicated disk block delivery to the target host
Source ESXi vMotion VMkernel
Local source IX vMotion interface
TCP 8000
Active memory and VM-state transfer during RAV switchover
Destination IX vMotion interface
Destination ESXi vMotion VMkernel
TCP 8000
Delivery of active memory and VM state to the destination host
HCX Manager
IX management interface
TCP 8123
HBR service communication used by Bulk and RAV workflows
Firewall Validation Rules
A firewall rule existing in a change ticket does not prove that the flow works. Validate from the endpoint and interface that will actually originate the traffic.
Use these methods:
Test HCX Manager-to-manager TCP 443 from the managers or an approved equivalent path.
Validate manager-to-vCenter and manager-to-ESXi TCP 443.
Use HCX Service Mesh diagnostics to validate IX and NE paths.
Use firewall flow logs or packet captures to confirm UDP 4500 in both directions.
Validate TCP 902 and TCP 8000 from the relevant ESXi VMkernel networks and IX interfaces.
Verify TCP 8123 is listening on the expected IX management address when Bulk or RAV diagnostics fail.
Confirm that NAT, proxy, or asymmetric routing does not alter the intended control path.
The following PowerShell example is useful for basic TCP checks from a Windows administration host. It does not validate UDP 4500 or prove that an ESXi VMkernel interface can reach an IX interface.
$Targets = @(
[pscustomobject]@{
ComputerName = “hcx-destination.corp.example”
Port = 443
Purpose = “Remote HCX Manager site pairing”
},
[pscustomobject]@{
ComputerName = “vcenter-source.corp.example”
Port = 443
Purpose = “Local vCenter API”
},
[pscustomobject]@{
ComputerName = “hcx-source.corp.example”
Port = 9443
Purpose = “HCX appliance management”
}
)
$Targets | ForEach-Object {
$result = Test-NetConnection `
-ComputerName $_.ComputerName `
-Port $_.Port `
-InformationLevel Detailed
[pscustomobject]@{
Target = $_.ComputerName
Port = $_.Port
Purpose = $_.Purpose
Resolved = $result.NameResolutionSucceeded
Reachable = $result.TcpTestSucceeded
RemoteIP = $result.RemoteAddress
}
}
Change the example FQDNs to match the environment. Successful execution should show correct DNS resolution and Reachable set to True.
A successful test from an administrator workstation still needs to be followed by HCX diagnostics and endpoint-specific flow validation.
Plan HCX Network Profile IP Pools
Network Profiles define the local networks and address pools used by HCX service appliances. Depending on the selected services and topology, profiles may be created for management, uplink, vMotion, and replication traffic.
The pool design should be completed before profiles are created in the HCX interface.
IP Pool Design Rules
Use statically reserved ranges, not an address block that another IPAM or DHCP process can allocate.
Use unique, non-overlapping pools for every profile at each site.
Do not configure different HCX traffic profiles with overlapping subnets.
Confirm the gateway is reachable from every address in the pool.
Confirm DNS and NTP reachability where the traffic type requires it.
Confirm the selected port group or segment has the intended VLAN, teaming, security, and MTU configuration.
Reserve enough addresses for the selected services, appliance scale, future Service Mesh growth, and any planned Network Extension high-availability design.
Keep spare addresses available for redeploy, upgrade, and replacement operations.
Record which team owns each subnet and firewall zone.
HCX 9.1 has a documented Service Mesh failure mode when Network Profile ranges overlap. The result can appear as a logical-router-port creation failure or an IX appliance in an unknown state.
Treat uniqueness as a hard design requirement, not a naming convention.
Example Source-Site Address Plan
The addresses below are examples only. Replace them with approved enterprise ranges.
Profile
Backing network
CIDR
Example pool
Gateway
MTU
HCX-SRC-MGMT
VLAN 510
10.50.10.0/27
10.50.10.10 to 10.50.10.20
10.50.10.1
1500
HCX-SRC-UPLINK
VLAN 520
10.50.20.0/27
10.50.20.10 to 10.50.20.20
10.50.20.1
1500
HCX-SRC-VMOTION
VLAN 530
10.50.30.0/27
10.50.30.10 to 10.50.30.20
10.50.30.1
9000
HCX-SRC-REPLICATION
VLAN 540
10.50.40.0/27
10.50.40.10 to 10.50.40.20
10.50.40.1
9000
Example Destination-Site Address Plan
Profile
Backing network
CIDR
Example pool
Gateway
MTU
HCX-DST-MGMT
VLAN 610
10.60.10.0/27
10.60.10.10 to 10.60.10.20
10.60.10.1
1500
HCX-DST-UPLINK
VLAN 620
10.60.20.0/27
10.60.20.10 to 10.60.20.20
10.60.20.1
1500
HCX-DST-VMOTION
VLAN 630
10.60.30.0/27
10.60.30.10 to 10.60.30.20
10.60.30.1
9000
HCX-DST-REPLICATION
VLAN 640
10.60.40.0/27
10.60.40.10 to 10.60.40.20
10.60.40.1
9000
Do not copy the MTU values blindly. Use jumbo frames only when every switch, routed hop, firewall, virtual switch, VMkernel interface, and HCX profile in the path supports the same effective packet size.
An inconsistent jumbo-frame design is worse than a consistent 1500-byte path.
Establish and Validate HCX Site Pairing
Site pairing creates the management relationship between the two HCX Managers. It is a control-plane dependency for Service Mesh creation and migration orchestration.
Site Pairing Readiness Checklist
Before pairing:
Confirm both managers resolve each other by the intended FQDN.
Confirm forward and reverse DNS records where required by local standards.
Confirm NTP synchronization and reasonable clock alignment.
Confirm the destination certificate is trusted or formally accepted according to policy.
Confirm TCP 443 in the required direction.
Confirm proxy exclusions for local vCenter, ESXi, NSX, IX, and NE endpoints.
Confirm the credential being used is permitted for remote-site pairing.
Confirm there is no asymmetric routing between managers.
Create the Site Pair
From the initiating HCX interface:
Open Site Pairing.
Add the remote HCX Manager using the approved FQDN.
Provide the authorized remote-site credentials.
Review the remote certificate and trust decision.
Complete the pairing workflow.
Confirm remote vCenter inventory becomes visible.
Refresh the interface and confirm the pairing remains healthy.
The exit gate is not merely that the site-pair object exists. Site pairing must remain healthy after refresh, and remote inventory must be usable from the migration and Interconnect workflows.
Site Pairing Failure Triage
If pairing fails:
Recheck DNS and NTP first.
Test TCP 443 from the initiating HCX Manager to the remote manager.
Review proxy configuration and bypass lists.
Confirm the remote certificate chain and FQDN match.
Check for asymmetric routing or unexpected NAT translation.
Review manager logs for timeout, authentication, or trust errors.
Capture packets from both manager sides if the TCP or TLS behavior remains ambiguous.
Do not proceed to Service Mesh creation while site pairing is unstable.
Existing extensions can sometimes continue forwarding traffic during a control-plane outage, but new configuration and migration workflows remain at risk.
Create HCX Network Profiles
Create one Network Profile for each local network that will be consumed by the selected HCX services.
Network Profile Configuration Sequence
For each profile:
Navigate to Interconnect and open Network Profiles.
Select the local vCenter inventory.
Select the intended distributed port group or NSX segment.
Enter the reserved IP range.
Enter the prefix length, gateway, DNS, and MTU values.
Assign the appropriate HCX traffic type.
Save the profile.
Confirm the configured range does not overlap another profile.
Confirm the gateway and required endpoints are reachable from that network.
Reconcile the saved values against the approved IP worksheet.
Use a consistent naming pattern that communicates site, purpose, and environment.
NP-SRC01-MGMT-HCX91
NP-SRC01-UPLINK-HCX91
NP-SRC01-VMOTION-HCX91
NP-SRC01-REPL-HCX91
NP-DST01-MGMT-HCX91
NP-DST01-UPLINK-HCX91
NP-DST01-VMOTION-HCX91
NP-DST01-REPL-HCX91
A name does not prove that the selected port group, VLAN, traffic type, or pool is correct. Export or capture the completed profile details before moving on.
Create HCX Compute Profiles
Compute Profiles define where HCX appliances can be deployed and which services they may provide. They bind together compute, storage, networks, service choices, and appliance resource intent.
Source Compute Profile
Create a source profile that includes only the resources intended for the migration program:
Source service cluster or approved deployment cluster
Appliance datastore or datastore set
Management Network Profile
Uplink Network Profile
vMotion Network Profile when required
Replication Network Profile when used
Migration services required by the runbook
Network Extension when the first test subnet will be extended
RAV only when RAV will be tested and supported
CPU and memory reservations configured through HCX
Destination Compute Profile
Create the destination profile with equivalent discipline:
Target service cluster or deployment cluster
Destination appliance datastore
Destination management, uplink, vMotion, and replication profiles
Target compute scope for VM placement
Required migration services
Required Network Extension services
HCX-managed resource reservations
Only the destination resources approved for migration
Broad profiles make implementation easy at first but weaken operational control. A migration operator should not accidentally see or select an unapproved cluster, datastore, or network simply because it exists in vCenter.
Compute Profile Review Questions
Before saving, ask:
Are the selected clusters compatible with the intended migration method?
Is appliance storage independent of a datastore entering maintenance?
Are the Network Profile mappings correct for this site?
Are CPU and memory reservations defined through HCX?
Is Network Extension enabled only where needed?
Is RAV enabled at both sites if RAV will be used?
Is the target compute scope narrow enough to prevent placement mistakes?
Will DRS rules protect appliance stability without blocking lifecycle operations?
Are service-account permissions sufficient for the selected switch and extension mode?
Deploy the HCX Service Mesh
A Service Mesh joins the source and destination Compute Profiles and deploys the appliances that provide the selected migration and Network Extension services.
Create the Service Mesh
Navigate to Interconnect and open Service Mesh.
Select the healthy site pair.
Select the source Compute Profile.
Select the destination Compute Profile.
Confirm the service list.
Confirm local and remote Network Profile mappings.
Select appliance scale and Network Extension mode appropriate to the design.
Apply a descriptive name such as SM-SRC01-DST01-HCX91-01.
Review the topology summary.
Start deployment.
Monitor appliance deployment, interface assignment, power state, and tunnel establishment.
Do not navigate away and assume the workflow will recover from every warning. A partially deployed mesh should be treated as failed until the underlying cause is understood.
Service Mesh Validation Gate
The Service Mesh is ready only when all required validation checks pass.
Validation
Pass condition
Site pairing
Healthy and remote inventory visible
IX appliances
Deployed, powered on, and healthy at both sites
NE appliances
Deployed and healthy when Network Extension is enabled
Appliance interfaces
Expected management, uplink, vMotion, and replication addresses assigned
Tunnels
Required tunnels established without unexplained flapping
Run Diagnostics
No unresolved red tests or blocked required ports
Transport Analytics
Underlay test completed and results recorded
Firewall evidence
UDP 4500 and required TCP flows observed or explicitly validated
Resource health
HCX-configured CPU and memory reservations present
Time and name resolution
DNS and NTP remain healthy from both managers
Run Service Mesh diagnostics and review Transport Analytics before migration. Save the output with the change record rather than treating it as a transient green screen.
Recognize the HCX 9.1 Port 8123 Failure
In HCX 9.1, Bulk and RAV migrations can fail even when firewall policy appears correct because the HBR service is not listening on TCP 8123 at the expected IX management address.
Indicators include:
Data Plane diagnostics report TCP 8123 blocked from HCX Manager to IX.
Bulk and RAV workflows fail with an HBR connection error.
The IX appliance does not show a listener on TCP 8123.
The HBR service is bound to localhost instead of the designated management address.
Do not immediately request another firewall exception if flow evidence shows the packet reaches the appliance. Confirm whether the service is listening and follow the current Broadcom remediation or support process.
If the documented service restart is used, perform it according to the Broadcom procedure on the affected peer appliances and rerun diagnostics before scheduling a migration.
Recognize the VDS Trunk Privilege Failure
HCX 9.1 deployments using a VMware Distributed Switch and the Multiple Extensions per vNIC Network Extension mode can fail while creating the trunk distributed port group if the HCX vCenter service role lacks the required permissions.
The required privileges identified in Broadcom’s current guidance are:
DVPortgroup.Create
DVPortgroup.Delete
If Service Mesh deployment reports permission denied while creating a trunk port group:
Review the HCX To vCenter Service Account Role.
Confirm the required distributed port-group create and delete privileges.
Apply the correction through the approved privilege-management process.
Retry Service Mesh deployment only after the role is verified.
The following PowerCLI command illustrates the documented role update. Confirm the role name and required privileges against the current Broadcom guidance before running it.
Connect-VIServer -Server “vcenter-source.corp.example”
$role = Get-VIRole -Name “HCX To vCenter Service Account Role”
$privileges = Get-VIPrivilege -Id @(
“DVPortgroup.Create”,
“DVPortgroup.Delete”
)
Set-VIRole -Role $role -AddPrivilege $privileges
Successful execution adds the missing privileges to the existing HCX role. Do not replace the role or grant broader administrative privileges as a shortcut.
Validate the HCX Underlay Before Extending a Network
A healthy Service Mesh proves that appliances deployed and established the required relationships. It does not prove that the underlay will sustain the migration window.
Run and record:
Service Mesh diagnostics
Transport Analytics for the intended uplinks
Latency, packet loss, jitter, and available throughput observations
MTU validation across the effective path
Firewall flow evidence for UDP 4500
CPU and memory health of IX and NE appliances
ESXi VMkernel reachability for the required migration paths
Recommended Bulk Migration concurrency produced by the underlay test
Use the measured result to set migration concurrency. Do not copy a concurrency number from another site or assume the WAN circuit’s purchased bandwidth is fully available to HCX during the migration window.
Extend One Test Network
Network Extension should be proven on a low-risk network before it is used for an application wave.
Do not begin with:
Management infrastructure
Directory services
Backup systems
Storage control networks
Cluster heartbeat networks
The most critical application subnet
Network Extension Readiness
Confirm:
The source port group or segment is supported for extension.
The destination backing network is correctly selected.
The same Layer 2 segment is not unintentionally bridged through another physical or virtual path.
The source gateway remains authoritative unless a separate routing cutover has been approved.
Firewalls and distributed firewall policies account for traffic crossing the HCX extension.
NSX SpoofGuard, MAC learning, and MAC discovery policies are compatible with HCX proxy behavior.
MTU is consistent across the source, HCX tunnel, and destination.
The rollback plan states whether the network will remain extended if the VM test fails.
Extend the Test Network
Open Network Extension in HCX.
Select the validated Service Mesh.
Select one source test network.
Map it to the correct destination network or gateway context.
Review extension mode and appliance placement.
Start the extension workflow.
Wait for the extension to report healthy.
Run Network Extension diagnostics.
Confirm the expected Layer 2 and routed behavior before migrating a VM.
Network Extension Test Matrix
Place one test VM at each site on the extended subnet where practical, then validate:
Test
Expected result
ARP between source and destination VMs
Bidirectional resolution without repeated failure
Same-subnet ICMP
Bidirectional communication where policy permits
Default gateway
Reachable from both sites
DNS
Forward and reverse resolution as required
Application ports
Required TCP or UDP flows succeed
North-south routing
Follows the documented gateway path
Packet size
No unexpected fragmentation or black-hole behavior
Security logging
Traffic is visible at the expected enforcement points
Watch for Strict NSX SpoofGuard Behavior
HCX 9.1 has a documented scenario where migrated and non-migrated VMs on the same extended NSX segment cannot communicate because a non-default SpoofGuard profile rejects proxied return traffic.
ARP may appear to reach the HCX Network Extension appliance but never reach the VM.
For a test segment, changing to the system default profile can help prove the cause. In production, do not weaken a required security policy without security-owner approval.
If the environment requires the strict profile:
Preserve packet-capture evidence.
Preserve the affected segment and profile names.
Do not make an unapproved security exception.
Open a Broadcom support case.
Review MAC learning and MAC discovery configuration for similar symptoms.
Prepare the First VM Move
The first move should not be the first time the team tests HCX end to end.
Use a two-stage approach:
First, move a powered-off synthetic VM to prove inventory placement, storage mapping, network mapping, and target power-on.
Then, use a low-risk Bulk Migration canary to prove replication, scheduled switchover, application validation, and rollback handling.
Select a Good Canary VM
Choose a VM with:
One standard network adapter
Standard virtual disks
No shared SCSI bus or multi-writer configuration
No Fault Tolerance
No DirectPath I/O
No attached installation media
No independent disks
Healthy VMware Tools
No active snapshot-based backup job during migration
A clear application owner
A short, repeatable validation plan
A known source recovery procedure
The canary should be representative enough to test the target environment but unimportant enough that a failed move does not create a business incident.
Capture a Source Baseline with PowerCLI
The following example records the source VM’s placement, networks, disks, and snapshots before migration. It is an evidence artifact, not an HCX migration command.
Connect-VIServer -Server “vcenter-source.corp.example”
$vmName = “HCX-CANARY-01”
$vm = Get-VM -Name $vmName -ErrorAction Stop
$baseline = [ordered]@{
CapturedAt = (Get-Date).ToString(“s”)
VM = [ordered]@{
Name = $vm.Name
PowerState = $vm.PowerState.ToString()
Cluster = $vm.VMHost.Parent.Name
Host = $vm.VMHost.Name
Folder = $vm.Folder.Name
Datastore = @(
$vm |
Get-Datastore |
Select-Object -ExpandProperty Name
)
}
NetworkAdapters = @(
Get-NetworkAdapter -VM $vm |
Select-Object Name, Type, NetworkName, MacAddress
)
HardDisks = @(
Get-HardDisk -VM $vm |
Select-Object Name, CapacityGB, StorageFormat, Filename
)
Snapshots = @(
Get-Snapshot -VM $vm -ErrorAction SilentlyContinue |
Select-Object Name, Created, SizeGB
)
}
$baseline |
ConvertTo-Json -Depth 6 |
Set-Content `
-Path “.$vmName-source-baseline.json” `
-Encoding UTF8
Change the vCenter FQDN and VM name. Successful execution creates a JSON baseline file for the exact source VM.
A duplicate VM name, insufficient permissions, or an inaccessible vCenter will stop the script and should be corrected before migration.
Define the Validation Script Before Cutover
Document the expected results for:
Guest IP address, prefix, and gateway
DNS servers and search suffixes
Domain or identity connectivity
Application listener ports
Database or middleware dependencies
Monitoring agent status
Backup-agent registration
Time synchronization
Endpoint-security health
Application transaction or synthetic test
For a Windows canary, an operator may use commands such as:
ipconfig /all
route print
Resolve-DnsName dc01.corp.example
Test-Connection 10.70.50.1 -Count 4
Test-NetConnection dc01.corp.example -Port 389
Test-NetConnection appdb01.corp.example -Port 1433
w32tm /query /status
Replace the sample gateway, domain controller, database host, and ports with the actual application dependencies.
Run the same commands before and after migration so the team compares evidence rather than relying on memory.
Execute a Synthetic Cold Migration
A powered-off synthetic VM provides a low-risk test of basic HCX migration plumbing.
Configure the Cold Migration
Shut down the disposable test VM.
Open Migrate Virtual Machines in HCX.
Select the validated remote site.
Select the synthetic VM.
Choose Cold Migration.
Select the approved target cluster or resource pool.
Select the approved datastore.
Select the target VM folder.
Map every network adapter.
Run HCX validation.
Resolve every blocking error.
Start the migration.
Confirm the target VM registers and powers on correctly.
Run the guest and network validation tests.
Remove the synthetic VM only after the evidence is recorded.
This test proves basic placement and network mapping. It does not prove Bulk Migration replication, scheduled switchover, or application consistency.
Execute the First Controlled Bulk Migration
Bulk Migration is a useful first production-like test because it exercises replication and a scheduled switchover while allowing the source VM to remain authoritative until cutover.
Configure the Migration
Open Migration in HCX.
Select Migrate Virtual Machines.
Select the validated remote site.
Select the canary VM.
Choose Bulk Migration.
Select the approved destination cluster or resource pool.
Select the approved destination datastore.
Select the destination VM folder.
Map every network adapter to the intended destination or extended network.
Configure the switchover window.
Enable Seed Checkpoint when appropriate.
Select a specific datastore when Seed Checkpoint is used.
Run HCX validation.
Resolve every blocking warning.
Start the migration.
Seed Checkpoint retains target disks after a failed or canceled Bulk or RAV workflow, which can reduce retransmission during a retry. Without it, rollback cleanup removes the transferred target disks.
This is a retry optimization, not a replacement for source recovery planning.
Monitor Replication and Switchover
During migration:
Monitor base synchronization and delta progress.
Watch target datastore capacity.
Confirm the Service Mesh remains healthy.
Do not restart HCX application services.
Do not restart vCenter services.
Do not manually power off the source VM after base synchronization.
Keep backup and snapshot workflows paused for the canary.
Preserve timestamps for every state transition.
Review migration events for unexpected pauses or repeated retries.
At switchover, allow more time than the best-case estimate. Final checksum, shutdown, delta synchronization, target instantiation, and guest boot can extend the outage beyond the nominal guest shutdown time.
Validate the Destination VM
After the target VM powers on:
Confirm one target VM exists and the source is not also active.
Confirm cluster, host, folder, datastore, and network placement.
Confirm MAC and IP behavior.
Run the predefined guest network tests.
Run application health and transaction tests.
Confirm logs, monitoring, endpoint security, and backup registration.
Check Network Extension health and tunnel stability.
Confirm no duplicate IP or MAC appears in the network.
Obtain application-owner acceptance.
Record the acceptance timestamp and approver.
Do not accept the migration because the HCX workflow says complete. HCX completion proves the migration workflow reached its terminal state. It does not prove that the application is healthy.
Rollback and Recovery
Rollback depends on how far the workflow progressed. Treat it as a state machine.
Before Switchover
If replication or validation fails before switchover:
Keep the source VM authoritative.
Cancel or pause according to the workflow state.
Preserve diagnostic evidence.
Correct the port, service, capacity, mapping, or profile issue.
Reuse Seed Checkpoint data when valid.
Rerun validation before retrying.
After Switchover but Before Acceptance
If the target VM is unhealthy:
Isolate the target VM from the network.
Power off the target VM.
Confirm the exact source VM state in vCenter and HCX.
Confirm the source network and gateway remain available.
If routing authority changed, restore the source-side path first.
Power on the source VM.
Run the original source validation tests.
Confirm application-owner recovery.
Preserve the target VM and logs until the incident owner approves cleanup.
Never power on source and target copies simultaneously while they share the same IP address, hostname, domain identity, or application identity.
Network Extension Rollback
If the VM migration fails but the extended network is healthy, leave the extension in place until the source workload is stable and the next action is approved.
Removing the extension during an active recovery adds another variable.
Unextend the test network only when:
No accepted target VM depends on the extension.
The source remains authoritative.
Routing and gateway ownership are understood.
The Network Extension removal workflow has its own validation and rollback plan.
If a gateway cutover has already occurred, restore routing authority before returning the workload to the source site.
Service Mesh Rollback
Do not delete a Service Mesh that is carrying an active migration or Network Extension.
If a new mesh deployment fails before use:
Capture the failed workflow details and appliance logs.
Correct Network Profile overlap, reachability, privilege, or resource issues.
Remove failed appliances only through the supported HCX workflow.
Redeploy the Service Mesh.
Rerun diagnostics and Transport Analytics.
HCX 9.1 Failure Patterns to Recognize Quickly
Symptom
Likely area
First evidence to collect
Operational response
Site pairing timeout
DNS, NTP, TCP 443, proxy, certificate, routing
Manager-to-manager TCP and TLS test
Correct the control-plane path before profile or mesh work
Logical router port creation failure
Overlapping Network Profile ranges
Profile CIDRs, pools, and appliance interface assignment
Create unique non-overlapping ranges and redeploy
IX diagnostics report TCP 8123 blocked
HBR service binding or an actual firewall block
Listener state, IX management binding, and flow logs
Distinguish a service-binding issue from a firewall issue
Service Mesh trunk port-group permission denied
HCX vCenter service role
Role privileges and selected Network Extension mode
Add only the documented missing privileges, then retry
IX or NE tunnel remains down
UDP 4500, routing, NAT, or MTU
Bidirectional flow logs, packet capture, and diagnostics
Correct the underlay before migration
RAV fails during replication
TCP 902 or replication path
ESXi VMkernel-to-IX connectivity
Correct the NFC path and rerun validation
RAV fails during switchover
TCP 8000 or vMotion path
vMotion VMkernel-to-IX connectivity
Correct the vMotion path before retry
Migrated and source-side VMs on the same subnet cannot communicate
NSX SpoofGuard or MAC learning
ARP capture at the VM, host, and NE appliance
Validate the security profile and preserve evidence
VM moves but application fails
Dependency mapping, security, DNS, or routing
Pre-migration and post-migration test comparison
Isolate the target and follow the source recovery procedure
Evidence to Collect Before Escalation
Do not wait until after cleanup to gather evidence. For a failed Service Mesh, Network Extension, or migration workflow, preserve:
Exact HCX Manager and fleet-appliance versions
Site-pair health state
Service Mesh health and diagnostic results
Transport Analytics results
Migration event timeline
Affected VM name and source host
Source and target vCenter task history
Network Profile and Compute Profile mappings
Firewall flow logs
Relevant packet captures
HCX support bundles
IX and NE appliance logs
Guest operating system validation output
Target datastore capacity
Exact error text and timestamp
Changes made during troubleshooting
Evidence should be captured before restarting services, deleting appliances, or cleaning the failed target VM.
Production Wave Readiness
A successful canary is not automatic approval for a large wave. Use the first move to update the design and operating assumptions.
Before scaling, confirm:
Measured throughput supports the proposed concurrency.
The switchover duration matches the application outage budget.
Target datastore headroom remains sufficient during replication.
Monitoring and backup ownership is clear at the destination.
Network Extension capacity and failure protection meet the wave design.
The rollback window is long enough to detect application defects.
Application dependency maps are complete.
The operations team can distinguish HCX completion from application acceptance.
The known HCX 9.1 issues relevant to the environment are documented in the change plan.
Each migration wave has a named decision owner.
The first migration should improve the runbook. Record what took longer than expected, which checks caught real defects, which alerts were noisy, and which teams were missing from the change bridge.
Conclusion
HCX 9.1 migration success depends less on clicking through the configuration wizard and more on preserving dependency order. Firewall paths must be proven from the actual endpoints. IP pools must be unique and operationally owned. Site pairing must remain healthy. Network Profiles must map to the correct networks. Compute Profiles must constrain placement and services. The Service Mesh must pass diagnostics before Network Extension or migration begins.
The first VM move should be a controlled experiment with evidence at every stage. A synthetic powered-off VM proves basic placement and network mapping. A low-risk Bulk Migration canary proves replication, scheduled switchover, application validation, and rollback handling. Neither should become authoritative at the destination until the defined acceptance criteria pass.
Most importantly, rollback must be designed before migration. The team should know which VM is authoritative, which gateway owns the subnet, when the target must be isolated, when the source can be powered on, and which artifacts must be preserved for support.
That discipline turns HCX from a set of migration features into a repeatable workload-mobility operating process.
External References
Broadcom TechDocs: Network Port and Protocol RequirementsCanonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/workload-mobility/vmware-hcx-user-guide-vcf-9-0/preparing-for-hcx-installations/network-port-and-protocol-requirements.html
Broadcom TechDocs: About the Service MeshCanonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/workload-mobility/vmware-hcx-user-guide-vcf-9-0/configuring-and-managing-the-hcx-interconnect/configuring-the-hcx-service-mesh/about-the-hcx-service-mesh.html
Broadcom TechDocs: Requirements for Network ExtensionCanonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/workload-mobility/vmware-hcx-user-guide-vcf-9-0/extending-networks-with-vmware-hcx/about-vmware-hcx-network-extension/requirements-for-network-extension.html
Broadcom TechDocs: Migrating Virtual Machines with VMware HCXCanonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/workload-mobility/vmware-hcx-user-guide-vcf-9-0/migrating-virtual-machines-with-vmware-hcx.html
Broadcom Knowledge Base: HCX Health Check and Best PracticesCanonical URL: https://knowledge.broadcom.com/external/article/371941/hcx-health-check-and-best-practices.html
Broadcom Knowledge Base: HCX Site Pairing Connectivity DiagnosticsCanonical URL: https://knowledge.broadcom.com/external/article/321629/hcx-site-pairing-connectivity-diagnosti.html
Broadcom Knowledge Base: HCX Bulk Migration operations and best practicesCanonical URL: https://knowledge.broadcom.com/external/article/323663/hcx-bulk-migration-operations-and-best.html
Broadcom Knowledge Base: HCX RAV and Bulk Migrations Fail with HttpHostConnectException on Port 8123Canonical URL: https://knowledge.broadcom.com/external/article/448301/hcx-rav-and-bulk-migrations-fail-with-ht.html
Broadcom Knowledge Base: Error: LR port creation failed during Service Mesh creation – HCXCanonical URL: https://knowledge.broadcom.com/external/article/446376/error-lr-port-creation-failed-during-ser.html
Broadcom Knowledge Base: Service Mesh creation fails during trunk portgroup creation due to missing priviliges on vCenterCanonical URL: https://knowledge.broadcom.com/external/article/440404/service-mesh-creation-fails-during-trunk.html
Broadcom Knowledge Base: VM connectivity issues on HCX extended networks due to NSX Global SpoofGuard profileCanonical URL: https://knowledge.broadcom.com/external/article/444071/vm-connectivity-issues-on-hcx-extended-n.html
How to Install and Update VCF PowerCLI 9.1 on Windows, macOS, and Linux
TL;DR PowerCLI is now distributed through the VCF.PowerCLI package. The older VMware.PowerCLI package is deprecated, although many familiar cmdlets and underlying VMware.*…
Next PostVCF 9.1 API with Postman: Generate a Collection from OpenAPI and Automate Your First WorkflowTL;DR VMware Cloud Foundation 9.1 provides an OpenAPI definition for SDDC Manager that can be imported into Postman to generate a structured collection of API requests. The generated collection is…
The post HCX 9.1 Migration Runbook: Network Profiles, Compute Profiles, Service Mesh, and First VM Move appeared first on Digital Thought Disruption.