HCX 9.1 Migration Runbook: Network Profiles, Compute Profiles, Service Mesh, and First VM Move

TL;DR

An HCX 9.1 migration should not begin with the Migrate Virtual Machines button. It should begin with a dependency-ordered runbook that proves routing, firewall policy, IP pools, site pairing, Network Profiles, Compute Profiles, Service Mesh health, and Network Extension behavior before a production workload is touched.

The practical sequence is:

1. Confirm supported versions, backups, DNS, NTP, certificates, routes, and firewall ownership.
2. Reserve unique, non-overlapping HCX appliance IP pools at both sites.
3. Establish and validate HCX site pairing over TCP 443.
4. Create Network Profiles for the required traffic types.
5. Create Compute Profiles that constrain placement and enable only the required services.
6. Deploy the Service Mesh and run diagnostics until every required appliance and tunnel is healthy.
7. Extend one low-risk test network and validate security, MTU, routing, and east-west behavior.
8. Move a disposable test VM, then a controlled Bulk Migration canary.
9. Keep the source authoritative until application, network, monitoring, and backup tests pass.
10. Roll back according to the actual migration state, not by improvising after cutover.

Introduction

HCX failures often appear late, even when their causes were introduced much earlier. A migration may fail at switchover because TCP 8000 is blocked. A Service Mesh may fail because two Network Profiles contain overlapping address ranges. A migrated VM may power on successfully but lose same-subnet connectivity because a strict NSX SpoofGuard profile rejects proxied return traffic.

The team then spends the maintenance window troubleshooting the final symptom instead of correcting the original design dependency.

That is why HCX configuration needs to be treated as a chain of evidence. Every stage should produce a clear pass condition before the next stage begins. Site pairing proves the managers can communicate. Network Profiles prove that the appliance interfaces have valid addressing and reachability. Compute Profiles prove placement, service scope, and resource intent. The Service Mesh proves that the data-plane appliances and tunnels can operate. Network Extension proves the Layer 2 path. The first VM move proves the complete migration workflow.

This runbook is written for architects, VMware administrators, network engineers, application owners, and change managers preparing an HCX 9.1 migration between vSphere-based sites. It focuses on the first controlled implementation path, not every HCX topology or migration service.

Objective and Exit Criteria

The objective is to build a working HCX 9.1 migration path between two vSphere-based sites and complete the first controlled VM move without making the destination authoritative until validation is complete.

The runbook is successful when all of the following are true:

• Both HCX Managers report healthy local platform connections.
• Site pairing is healthy and remote inventory is visible.
• Network Profile address pools are unique, reserved, reachable, and large enough for the selected design.
• Compute Profiles contain the intended clusters, datastores, networks, services, and resource reservations.
• Service Mesh appliances deploy successfully and report a healthy state.
• HCX diagnostics report no unexplained blocked paths.
• The selected test network extends successfully.
• A test VM can communicate across the extended network.
• The first canary migration completes and passes technical and application validation.
• The source recovery path remains documented and executable until acceptance.

Scope and Assumptions

This runbook assumes:

• HCX 9.1 Managers are deployed or ready to be deployed at both sites.
• Both sites are vSphere-based and are within the current HCX interoperability and support boundaries.
• The required HCX entitlement is available for the services being enabled.
• DNS, NTP, certificate trust, vCenter connectivity, and administrative access are available.
• The network team can validate firewall flows from the actual HCX appliance and VMkernel networks, not only from an administrator workstation.
• The first production-like test will use a low-risk VM with a defined owner and validation plan.
• Network Extension will initially be limited to one test VLAN or NSX segment.

This runbook does not cover OS Assisted Migration, disaster recovery orchestration, cloud-provider-specific restrictions, Mobility Optimized Networking design, or a full subnet gateway cutover.

The HCX Dependency Order

The most important design principle is that HCX is built in dependency order. A green object at one layer does not prove the next layer is healthy, but a failure at an early layer can invalidate everything above it.

The diagram below shows the sequence that should control the change plan.

The operational lesson is straightforward: do not troubleshoot a migration workflow until the Service Mesh is healthy, and do not troubleshoot the Service Mesh until its profiles, routes, pools, permissions, and firewall paths are proven.

Pre-Change Safety Checks

Confirm the Version and Support Baseline

Record the exact HCX Manager and fleet-appliance versions at both sites. Confirm compatibility with the local vCenter, ESXi, NSX, and any cloud-provider endpoint.

Review current HCX 9.1 release information and known issues before the maintenance window. A generic HCX procedure may not account for a version-specific behavior.

Do not assume that two sites reporting “9.1” are on the same patch level. Record:

• Complete HCX Manager version and build
• IX and NE appliance versions
• vCenter and ESXi versions
• NSX version when applicable
• Approved target baseline
• Research and validation date

Back Up the HCX Managers

Configure and verify an HCX Manager backup before modifying site pairing, profiles, or Service Mesh objects.

The change record should include:

• Last successful backup timestamp
• Backup destination and retention
• Restore credentials and ownership
• Exact HCX build represented by the backup
• Recovery decision owner
• Confirmation that the backup is recent enough for the current HCX inventory

A VM snapshot is not a substitute for the supported HCX Manager backup workflow.

Freeze Conflicting Operations

Before deploying or changing the Service Mesh:

• Pause vCenter, ESXi, NSX, and HCX lifecycle changes.
• Pause backup jobs that create snapshots on the canary VM.
• Avoid datastore maintenance affecting HCX appliances.
• Avoid DRS or automation rules that repeatedly move HCX appliances.
• Confirm there are no existing migrations in a switchover state.
• Confirm no Network Extension changes are in progress.
• Confirm no Service Mesh resynchronization or redeployment is active.

Assign Operational Ownership

Do not enter the first migration window without named owners who can make decisions. A troubleshooting bridge full of observers but no decision authority will extend the outage.

HCX 9.1 Firewall Ports and Traffic Paths

The official HCX 9.1 port matrix remains the governing source. The table below is a runbook shortlist for the paths most likely to affect site pairing, Service Mesh deployment, Network Extension, Bulk Migration, and Replication Assisted vMotion.

It is not a replacement for the complete Broadcom port and protocol requirements.

Firewall Validation Rules

A firewall rule existing in a change ticket does not prove that the flow works. Validate from the endpoint and interface that will actually originate the traffic.

Use these methods:

• Test HCX Manager-to-manager TCP 443 from the managers or an approved equivalent path.
• Validate manager-to-vCenter and manager-to-ESXi TCP 443.
• Use HCX Service Mesh diagnostics to validate IX and NE paths.
• Use firewall flow logs or packet captures to confirm UDP 4500 in both directions.
• Validate TCP 902 and TCP 8000 from the relevant ESXi VMkernel networks and IX interfaces.
• Verify TCP 8123 is listening on the expected IX management address when Bulk or RAV diagnostics fail.
• Confirm that NAT, proxy, or asymmetric routing does not alter the intended control path.

The following PowerShell example is useful for basic TCP checks from a Windows administration host. It does not validate UDP 4500 or prove that an ESXi VMkernel interface can reach an IX interface.

$Targets = @(
    [pscustomobject]@{
        ComputerName = "hcx-destination.corp.example"
        Port         = 443
        Purpose      = "Remote HCX Manager site pairing"
    },
    [pscustomobject]@{
        ComputerName = "vcenter-source.corp.example"
        Port         = 443
        Purpose      = "Local vCenter API"
    },
    [pscustomobject]@{
        ComputerName = "hcx-source.corp.example"
        Port         = 9443
        Purpose      = "HCX appliance management"
    }
)

$Targets | ForEach-Object {
    $result = Test-NetConnection `
        -ComputerName $_.ComputerName `
        -Port $_.Port `
        -InformationLevel Detailed

    [pscustomobject]@{
        Target    = $_.ComputerName
        Port      = $_.Port
        Purpose   = $_.Purpose
        Resolved  = $result.NameResolutionSucceeded
        Reachable = $result.TcpTestSucceeded
        RemoteIP  = $result.RemoteAddress
    }
}

Change the example FQDNs to match the environment. Successful execution should show correct DNS resolution and Reachable set to True.

A successful test from an administrator workstation still needs to be followed by HCX diagnostics and endpoint-specific flow validation.

Plan HCX Network Profile IP Pools

Network Profiles define the local networks and address pools used by HCX service appliances. Depending on the selected services and topology, profiles may be created for management, uplink, vMotion, and replication traffic.

The pool design should be completed before profiles are created in the HCX interface.

IP Pool Design Rules

• Use statically reserved ranges, not an address block that another IPAM or DHCP process can allocate.
• Use unique, non-overlapping pools for every profile at each site.
• Do not configure different HCX traffic profiles with overlapping subnets.
• Confirm the gateway is reachable from every address in the pool.
• Confirm DNS and NTP reachability where the traffic type requires it.
• Confirm the selected port group or segment has the intended VLAN, teaming, security, and MTU configuration.
• Reserve enough addresses for the selected services, appliance scale, future Service Mesh growth, and any planned Network Extension high-availability design.
• Keep spare addresses available for redeploy, upgrade, and replacement operations.
• Record which team owns each subnet and firewall zone.

HCX 9.1 has a documented Service Mesh failure mode when Network Profile ranges overlap. The result can appear as a logical-router-port creation failure or an IX appliance in an unknown state.

Treat uniqueness as a hard design requirement, not a naming convention.

Example Source-Site Address Plan

The addresses below are examples only. Replace them with approved enterprise ranges.

Example Destination-Site Address Plan

Do not copy the MTU values blindly. Use jumbo frames only when every switch, routed hop, firewall, virtual switch, VMkernel interface, and HCX profile in the path supports the same effective packet size.

An inconsistent jumbo-frame design is worse than a consistent 1500-byte path.

Establish and Validate HCX Site Pairing

Site pairing creates the management relationship between the two HCX Managers. It is a control-plane dependency for Service Mesh creation and migration orchestration.

Site Pairing Readiness Checklist

Before pairing:

• Confirm both managers resolve each other by the intended FQDN.
• Confirm forward and reverse DNS records where required by local standards.
• Confirm NTP synchronization and reasonable clock alignment.
• Confirm the destination certificate is trusted or formally accepted according to policy.
• Confirm TCP 443 in the required direction.
• Confirm proxy exclusions for local vCenter, ESXi, NSX, IX, and NE endpoints.
• Confirm the credential being used is permitted for remote-site pairing.
• Confirm there is no asymmetric routing between managers.

Create the Site Pair

From the initiating HCX interface:

1. Open Site Pairing.
2. Add the remote HCX Manager using the approved FQDN.
3. Provide the authorized remote-site credentials.
4. Review the remote certificate and trust decision.
5. Complete the pairing workflow.
6. Confirm remote vCenter inventory becomes visible.
7. Refresh the interface and confirm the pairing remains healthy.

The exit gate is not merely that the site-pair object exists. Site pairing must remain healthy after refresh, and remote inventory must be usable from the migration and Interconnect workflows.

Site Pairing Failure Triage

If pairing fails:

1. Recheck DNS and NTP first.
2. Test TCP 443 from the initiating HCX Manager to the remote manager.
3. Review proxy configuration and bypass lists.
4. Confirm the remote certificate chain and FQDN match.
5. Check for asymmetric routing or unexpected NAT translation.
6. Review manager logs for timeout, authentication, or trust errors.
7. Capture packets from both manager sides if the TCP or TLS behavior remains ambiguous.

Do not proceed to Service Mesh creation while site pairing is unstable.

Existing extensions can sometimes continue forwarding traffic during a control-plane outage, but new configuration and migration workflows remain at risk.

Create HCX Network Profiles

Create one Network Profile for each local network that will be consumed by the selected HCX services.

Network Profile Configuration Sequence

For each profile:

1. Navigate to Interconnect and open Network Profiles.
2. Select the local vCenter inventory.
3. Select the intended distributed port group or NSX segment.
4. Enter the reserved IP range.
5. Enter the prefix length, gateway, DNS, and MTU values.
6. Assign the appropriate HCX traffic type.
7. Save the profile.
8. Confirm the configured range does not overlap another profile.
9. Confirm the gateway and required endpoints are reachable from that network.
10. Reconcile the saved values against the approved IP worksheet.

Use a consistent naming pattern that communicates site, purpose, and environment.

NP-SRC01-MGMT-HCX91
NP-SRC01-UPLINK-HCX91
NP-SRC01-VMOTION-HCX91
NP-SRC01-REPL-HCX91

NP-DST01-MGMT-HCX91
NP-DST01-UPLINK-HCX91
NP-DST01-VMOTION-HCX91
NP-DST01-REPL-HCX91

A name does not prove that the selected port group, VLAN, traffic type, or pool is correct. Export or capture the completed profile details before moving on.

Create HCX Compute Profiles

Compute Profiles define where HCX appliances can be deployed and which services they may provide. They bind together compute, storage, networks, service choices, and appliance resource intent.

Source Compute Profile

Create a source profile that includes only the resources intended for the migration program:

• Source service cluster or approved deployment cluster
• Appliance datastore or datastore set
• Management Network Profile
• Uplink Network Profile
• vMotion Network Profile when required
• Replication Network Profile when used
• Migration services required by the runbook
• Network Extension when the first test subnet will be extended
• RAV only when RAV will be tested and supported
• CPU and memory reservations configured through HCX

Destination Compute Profile

Create the destination profile with equivalent discipline:

• Target service cluster or deployment cluster
• Destination appliance datastore
• Destination management, uplink, vMotion, and replication profiles
• Target compute scope for VM placement
• Required migration services
• Required Network Extension services
• HCX-managed resource reservations
• Only the destination resources approved for migration

Broad profiles make implementation easy at first but weaken operational control. A migration operator should not accidentally see or select an unapproved cluster, datastore, or network simply because it exists in vCenter.

Compute Profile Review Questions

Before saving, ask:

• Are the selected clusters compatible with the intended migration method?
• Is appliance storage independent of a datastore entering maintenance?
• Are the Network Profile mappings correct for this site?
• Are CPU and memory reservations defined through HCX?
• Is Network Extension enabled only where needed?
• Is RAV enabled at both sites if RAV will be used?
• Is the target compute scope narrow enough to prevent placement mistakes?
• Will DRS rules protect appliance stability without blocking lifecycle operations?
• Are service-account permissions sufficient for the selected switch and extension mode?

Deploy the HCX Service Mesh

A Service Mesh joins the source and destination Compute Profiles and deploys the appliances that provide the selected migration and Network Extension services.

Create the Service Mesh

1. Navigate to Interconnect and open Service Mesh.
2. Select the healthy site pair.
3. Select the source Compute Profile.
4. Select the destination Compute Profile.
5. Confirm the service list.
6. Confirm local and remote Network Profile mappings.
7. Select appliance scale and Network Extension mode appropriate to the design.
8. Apply a descriptive name such as SM-SRC01-DST01-HCX91-01.
9. Review the topology summary.
10. Start deployment.
11. Monitor appliance deployment, interface assignment, power state, and tunnel establishment.

Do not navigate away and assume the workflow will recover from every warning. A partially deployed mesh should be treated as failed until the underlying cause is understood.

Service Mesh Validation Gate

The Service Mesh is ready only when all required validation checks pass.

Run Service Mesh diagnostics and review Transport Analytics before migration. Save the output with the change record rather than treating it as a transient green screen.

Recognize the HCX 9.1 Port 8123 Failure

In HCX 9.1, Bulk and RAV migrations can fail even when firewall policy appears correct because the HBR service is not listening on TCP 8123 at the expected IX management address.

Indicators include:

• Data Plane diagnostics report TCP 8123 blocked from HCX Manager to IX.
• Bulk and RAV workflows fail with an HBR connection error.
• The IX appliance does not show a listener on TCP 8123.
• The HBR service is bound to localhost instead of the designated management address.

Do not immediately request another firewall exception if flow evidence shows the packet reaches the appliance. Confirm whether the service is listening and follow the current Broadcom remediation or support process.

If the documented service restart is used, perform it according to the Broadcom procedure on the affected peer appliances and rerun diagnostics before scheduling a migration.

Recognize the VDS Trunk Privilege Failure

HCX 9.1 deployments using a VMware Distributed Switch and the Multiple Extensions per vNIC Network Extension mode can fail while creating the trunk distributed port group if the HCX vCenter service role lacks the required permissions.

The required privileges identified in Broadcom’s current guidance are:

DVPortgroup.Create
DVPortgroup.Delete

If Service Mesh deployment reports permission denied while creating a trunk port group:

• Review the HCX To vCenter Service Account Role.
• Confirm the required distributed port-group create and delete privileges.
• Apply the correction through the approved privilege-management process.
• Retry Service Mesh deployment only after the role is verified.

The following PowerCLI command illustrates the documented role update. Confirm the role name and required privileges against the current Broadcom guidance before running it.

Connect-VIServer -Server "vcenter-source.corp.example"

$role = Get-VIRole -Name "HCX To vCenter Service Account Role"

$privileges = Get-VIPrivilege -Id @(
    "DVPortgroup.Create",
    "DVPortgroup.Delete"
)

Set-VIRole -Role $role -AddPrivilege $privileges

Successful execution adds the missing privileges to the existing HCX role. Do not replace the role or grant broader administrative privileges as a shortcut.

Validate the HCX Underlay Before Extending a Network

A healthy Service Mesh proves that appliances deployed and established the required relationships. It does not prove that the underlay will sustain the migration window.

Run and record:

• Service Mesh diagnostics
• Transport Analytics for the intended uplinks
• Latency, packet loss, jitter, and available throughput observations
• MTU validation across the effective path
• Firewall flow evidence for UDP 4500
• CPU and memory health of IX and NE appliances
• ESXi VMkernel reachability for the required migration paths
• Recommended Bulk Migration concurrency produced by the underlay test

Use the measured result to set migration concurrency. Do not copy a concurrency number from another site or assume the WAN circuit’s purchased bandwidth is fully available to HCX during the migration window.

Extend One Test Network

Network Extension should be proven on a low-risk network before it is used for an application wave.

Do not begin with:

• Management infrastructure
• Directory services
• Backup systems
• Storage control networks
• Cluster heartbeat networks
• The most critical application subnet

Network Extension Readiness

Confirm:

• The source port group or segment is supported for extension.
• The destination backing network is correctly selected.
• The same Layer 2 segment is not unintentionally bridged through another physical or virtual path.
• The source gateway remains authoritative unless a separate routing cutover has been approved.
• Firewalls and distributed firewall policies account for traffic crossing the HCX extension.
• NSX SpoofGuard, MAC learning, and MAC discovery policies are compatible with HCX proxy behavior.
• MTU is consistent across the source, HCX tunnel, and destination.
• The rollback plan states whether the network will remain extended if the VM test fails.

Extend the Test Network

1. Open Network Extension in HCX.
2. Select the validated Service Mesh.
3. Select one source test network.
4. Map it to the correct destination network or gateway context.
5. Review extension mode and appliance placement.
6. Start the extension workflow.
7. Wait for the extension to report healthy.
8. Run Network Extension diagnostics.
9. Confirm the expected Layer 2 and routed behavior before migrating a VM.

Network Extension Test Matrix

Place one test VM at each site on the extended subnet where practical, then validate:

Watch for Strict NSX SpoofGuard Behavior

HCX 9.1 has a documented scenario where migrated and non-migrated VMs on the same extended NSX segment cannot communicate because a non-default SpoofGuard profile rejects proxied return traffic.

ARP may appear to reach the HCX Network Extension appliance but never reach the VM.

For a test segment, changing to the system default profile can help prove the cause. In production, do not weaken a required security policy without security-owner approval.

If the environment requires the strict profile:

• Preserve packet-capture evidence.
• Preserve the affected segment and profile names.
• Do not make an unapproved security exception.
• Open a Broadcom support case.
• Review MAC learning and MAC discovery configuration for similar symptoms.

Prepare the First VM Move

The first move should not be the first time the team tests HCX end to end.

Use a two-stage approach:

• First, move a powered-off synthetic VM to prove inventory placement, storage mapping, network mapping, and target power-on.
• Then, use a low-risk Bulk Migration canary to prove replication, scheduled switchover, application validation, and rollback handling.

Select a Good Canary VM

Choose a VM with:

• One standard network adapter
• Standard virtual disks
• No shared SCSI bus or multi-writer configuration
• No Fault Tolerance
• No DirectPath I/O
• No attached installation media
• No independent disks
• Healthy VMware Tools
• No active snapshot-based backup job during migration
• A clear application owner
• A short, repeatable validation plan
• A known source recovery procedure

The canary should be representative enough to test the target environment but unimportant enough that a failed move does not create a business incident.

Capture a Source Baseline with PowerCLI

The following example records the source VM’s placement, networks, disks, and snapshots before migration. It is an evidence artifact, not an HCX migration command.

Connect-VIServer -Server "vcenter-source.corp.example"

$vmName = "HCX-CANARY-01"
$vm = Get-VM -Name $vmName -ErrorAction Stop

$baseline = [ordered]@{
    CapturedAt = (Get-Date).ToString("s")
    VM         = [ordered]@{
        Name       = $vm.Name
        PowerState = $vm.PowerState.ToString()
        Cluster    = $vm.VMHost.Parent.Name
        Host       = $vm.VMHost.Name
        Folder     = $vm.Folder.Name
        Datastore  = @(
            $vm |
                Get-Datastore |
                Select-Object -ExpandProperty Name
        )
    }
    NetworkAdapters = @(
        Get-NetworkAdapter -VM $vm |
            Select-Object Name, Type, NetworkName, MacAddress
    )
    HardDisks = @(
        Get-HardDisk -VM $vm |
            Select-Object Name, CapacityGB, StorageFormat, Filename
    )
    Snapshots = @(
        Get-Snapshot -VM $vm -ErrorAction SilentlyContinue |
            Select-Object Name, Created, SizeGB
    )
}

$baseline |
    ConvertTo-Json -Depth 6 |
    Set-Content `
        -Path ".$vmName-source-baseline.json" `
        -Encoding UTF8

Change the vCenter FQDN and VM name. Successful execution creates a JSON baseline file for the exact source VM.

A duplicate VM name, insufficient permissions, or an inaccessible vCenter will stop the script and should be corrected before migration.

Define the Validation Script Before Cutover

Document the expected results for:

• Guest IP address, prefix, and gateway
• DNS servers and search suffixes
• Domain or identity connectivity
• Application listener ports
• Database or middleware dependencies
• Monitoring agent status
• Backup-agent registration
• Time synchronization
• Endpoint-security health
• Application transaction or synthetic test

For a Windows canary, an operator may use commands such as:

ipconfig /all
route print

Resolve-DnsName dc01.corp.example

Test-Connection 10.70.50.1 -Count 4

Test-NetConnection dc01.corp.example -Port 389

Test-NetConnection appdb01.corp.example -Port 1433

w32tm /query /status

Replace the sample gateway, domain controller, database host, and ports with the actual application dependencies.

Run the same commands before and after migration so the team compares evidence rather than relying on memory.

Execute a Synthetic Cold Migration

A powered-off synthetic VM provides a low-risk test of basic HCX migration plumbing.

Configure the Cold Migration

1. Shut down the disposable test VM.
2. Open Migrate Virtual Machines in HCX.
3. Select the validated remote site.
4. Select the synthetic VM.
5. Choose Cold Migration.
6. Select the approved target cluster or resource pool.
7. Select the approved datastore.
8. Select the target VM folder.
9. Map every network adapter.
10. Run HCX validation.
11. Resolve every blocking error.
12. Start the migration.
13. Confirm the target VM registers and powers on correctly.
14. Run the guest and network validation tests.
15. Remove the synthetic VM only after the evidence is recorded.

This test proves basic placement and network mapping. It does not prove Bulk Migration replication, scheduled switchover, or application consistency.

Execute the First Controlled Bulk Migration

Bulk Migration is a useful first production-like test because it exercises replication and a scheduled switchover while allowing the source VM to remain authoritative until cutover.

Configure the Migration

1. Open Migration in HCX.
2. Select Migrate Virtual Machines.
3. Select the validated remote site.
4. Select the canary VM.
5. Choose Bulk Migration.
6. Select the approved destination cluster or resource pool.
7. Select the approved destination datastore.
8. Select the destination VM folder.
9. Map every network adapter to the intended destination or extended network.
10. Configure the switchover window.
11. Enable Seed Checkpoint when appropriate.
12. Select a specific datastore when Seed Checkpoint is used.
13. Run HCX validation.
14. Resolve every blocking warning.
15. Start the migration.

Seed Checkpoint retains target disks after a failed or canceled Bulk or RAV workflow, which can reduce retransmission during a retry. Without it, rollback cleanup removes the transferred target disks.

This is a retry optimization, not a replacement for source recovery planning.

Monitor Replication and Switchover

During migration:

• Monitor base synchronization and delta progress.
• Watch target datastore capacity.
• Confirm the Service Mesh remains healthy.
• Do not restart HCX application services.
• Do not restart vCenter services.
• Do not manually power off the source VM after base synchronization.
• Keep backup and snapshot workflows paused for the canary.
• Preserve timestamps for every state transition.
• Review migration events for unexpected pauses or repeated retries.

At switchover, allow more time than the best-case estimate. Final checksum, shutdown, delta synchronization, target instantiation, and guest boot can extend the outage beyond the nominal guest shutdown time.

Validate the Destination VM

After the target VM powers on:

1. Confirm one target VM exists and the source is not also active.
2. Confirm cluster, host, folder, datastore, and network placement.
3. Confirm MAC and IP behavior.
4. Run the predefined guest network tests.
5. Run application health and transaction tests.
6. Confirm logs, monitoring, endpoint security, and backup registration.
7. Check Network Extension health and tunnel stability.
8. Confirm no duplicate IP or MAC appears in the network.
9. Obtain application-owner acceptance.
10. Record the acceptance timestamp and approver.

Do not accept the migration because the HCX workflow says complete. HCX completion proves the migration workflow reached its terminal state. It does not prove that the application is healthy.

Rollback and Recovery

Rollback depends on how far the workflow progressed. Treat it as a state machine.

Before Switchover

If replication or validation fails before switchover:

• Keep the source VM authoritative.
• Cancel or pause according to the workflow state.
• Preserve diagnostic evidence.
• Correct the port, service, capacity, mapping, or profile issue.
• Reuse Seed Checkpoint data when valid.
• Rerun validation before retrying.

After Switchover but Before Acceptance

If the target VM is unhealthy:

1. Isolate the target VM from the network.
2. Power off the target VM.
3. Confirm the exact source VM state in vCenter and HCX.
4. Confirm the source network and gateway remain available.
5. If routing authority changed, restore the source-side path first.
6. Power on the source VM.
7. Run the original source validation tests.
8. Confirm application-owner recovery.
9. Preserve the target VM and logs until the incident owner approves cleanup.

Never power on source and target copies simultaneously while they share the same IP address, hostname, domain identity, or application identity.

Network Extension Rollback

If the VM migration fails but the extended network is healthy, leave the extension in place until the source workload is stable and the next action is approved.

Removing the extension during an active recovery adds another variable.

Unextend the test network only when:

• No accepted target VM depends on the extension.
• The source remains authoritative.
• Routing and gateway ownership are understood.
• The Network Extension removal workflow has its own validation and rollback plan.

If a gateway cutover has already occurred, restore routing authority before returning the workload to the source site.

Service Mesh Rollback

Do not delete a Service Mesh that is carrying an active migration or Network Extension.

If a new mesh deployment fails before use:

• Capture the failed workflow details and appliance logs.
• Correct Network Profile overlap, reachability, privilege, or resource issues.
• Remove failed appliances only through the supported HCX workflow.
• Redeploy the Service Mesh.
• Rerun diagnostics and Transport Analytics.

HCX 9.1 Failure Patterns to Recognize Quickly

Evidence to Collect Before Escalation

Do not wait until after cleanup to gather evidence. For a failed Service Mesh, Network Extension, or migration workflow, preserve:

• Exact HCX Manager and fleet-appliance versions
• Site-pair health state
• Service Mesh health and diagnostic results
• Transport Analytics results
• Migration event timeline
• Affected VM name and source host
• Source and target vCenter task history
• Network Profile and Compute Profile mappings
• Firewall flow logs
• Relevant packet captures
• HCX support bundles
• IX and NE appliance logs
• Guest operating system validation output
• Target datastore capacity
• Exact error text and timestamp
• Changes made during troubleshooting

Evidence should be captured before restarting services, deleting appliances, or cleaning the failed target VM.

Production Wave Readiness

A successful canary is not automatic approval for a large wave. Use the first move to update the design and operating assumptions.

Before scaling, confirm:

• Measured throughput supports the proposed concurrency.
• The switchover duration matches the application outage budget.
• Target datastore headroom remains sufficient during replication.
• Monitoring and backup ownership is clear at the destination.
• Network Extension capacity and failure protection meet the wave design.
• The rollback window is long enough to detect application defects.
• Application dependency maps are complete.
• The operations team can distinguish HCX completion from application acceptance.
• The known HCX 9.1 issues relevant to the environment are documented in the change plan.
• Each migration wave has a named decision owner.

The first migration should improve the runbook. Record what took longer than expected, which checks caught real defects, which alerts were noisy, and which teams were missing from the change bridge.

Conclusion

HCX 9.1 migration success depends less on clicking through the configuration wizard and more on preserving dependency order. Firewall paths must be proven from the actual endpoints. IP pools must be unique and operationally owned. Site pairing must remain healthy. Network Profiles must map to the correct networks. Compute Profiles must constrain placement and services. The Service Mesh must pass diagnostics before Network Extension or migration begins.

The first VM move should be a controlled experiment with evidence at every stage. A synthetic powered-off VM proves basic placement and network mapping. A low-risk Bulk Migration canary proves replication, scheduled switchover, application validation, and rollback handling. Neither should become authoritative at the destination until the defined acceptance criteria pass.

Most importantly, rollback must be designed before migration. The team should know which VM is authoritative, which gateway owns the subnet, when the target must be isolated, when the source can be powered on, and which artifacts must be preserved for support.

That discipline turns HCX from a set of migration features into a repeatable workload-mobility operating process.

External References

• Broadcom TechDocs: Network Port and Protocol Requirements
  Canonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/workload-mobility/vmware-hcx-user-guide-vcf-9-0/preparing-for-hcx-installations/network-port-and-protocol-requirements.html
• Broadcom TechDocs: About the Service Mesh
  Canonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/workload-mobility/vmware-hcx-user-guide-vcf-9-0/configuring-and-managing-the-hcx-interconnect/configuring-the-hcx-service-mesh/about-the-hcx-service-mesh.html
• Broadcom TechDocs: Requirements for Network Extension
  Canonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/workload-mobility/vmware-hcx-user-guide-vcf-9-0/extending-networks-with-vmware-hcx/about-vmware-hcx-network-extension/requirements-for-network-extension.html
• Broadcom TechDocs: Migrating Virtual Machines with VMware HCX
  Canonical URL: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/workload-mobility/vmware-hcx-user-guide-vcf-9-0/migrating-virtual-machines-with-vmware-hcx.html
• Broadcom Knowledge Base: HCX Health Check and Best Practices
  Canonical URL: https://knowledge.broadcom.com/external/article/371941/hcx-health-check-and-best-practices.html
• Broadcom Knowledge Base: HCX Site Pairing Connectivity Diagnostics
  Canonical URL: https://knowledge.broadcom.com/external/article/321629/hcx-site-pairing-connectivity-diagnosti.html
• Broadcom Knowledge Base: HCX Bulk Migration operations and best practices
  Canonical URL: https://knowledge.broadcom.com/external/article/323663/hcx-bulk-migration-operations-and-best.html
• Broadcom Knowledge Base: HCX RAV and Bulk Migrations Fail with HttpHostConnectException on Port 8123
  Canonical URL: https://knowledge.broadcom.com/external/article/448301/hcx-rav-and-bulk-migrations-fail-with-ht.html
• Broadcom Knowledge Base: Error: LR port creation failed during Service Mesh creation – HCX
  Canonical URL: https://knowledge.broadcom.com/external/article/446376/error-lr-port-creation-failed-during-ser.html
• Broadcom Knowledge Base: Service Mesh creation fails during trunk portgroup creation due to missing priviliges on vCenter
  Canonical URL: https://knowledge.broadcom.com/external/article/440404/service-mesh-creation-fails-during-trunk.html
• Broadcom Knowledge Base: VM connectivity issues on HCX extended networks due to NSX Global SpoofGuard profile
  Canonical URL: https://knowledge.broadcom.com/external/article/444071/vm-connectivity-issues-on-hcx-extended-n.html

The post HCX 9.1 Migration Runbook: Network Profiles, Compute Profiles, Service Mesh, and First VM Move appeared first on Digital Thought Disruption.