Nutanix CVM Troubleshooting Command Reference: Stargate, Curator, Cassandra, Genesis, and Cluster Health

TL;DR

A Nutanix CVM incident should be investigated as an evidence chain, not as a race to restart services. Begin with cluster status, compare local and cluster-wide service state, check resiliency, identify the service that owns the symptom, collect its logs, and use targeted NCC checks to validate what you found.

This reference is intentionally conservative. Most commands are read-only and suitable for initial triage. Service restarts, cluster start or stop operations, Cassandra ring changes, forced Curator activity, CVM maintenance mode, and manual file deletion are escalation actions that should follow Nutanix guidance and an approved change plan.

Introduction

A Controller VM problem rarely presents itself with a clean label that says, “Stargate is the root cause” or “Cassandra membership is unhealthy.” Operators usually see an indirect symptom first: Prism is unavailable, VM storage latency increases, capacity is not reclaimed, an NCC check fails, a service repeatedly restarts, or a cluster does not fully recover after maintenance.

That is why a useful CVM command reference must be organized around four questions:

• What symptom is the operator observing?
• Which service is most likely involved?
• What evidence confirms or disproves that theory?
• At what point should local troubleshooting stop and escalation begin?

This companion article builds on the architectural view of the Nutanix CVM and turns it into an operational runbook. The goal is not to document every internal utility. The goal is to give administrators a safe, repeatable sequence for Stargate, Curator, Cassandra, Genesis, and overall cluster health.

Operational Scope and Safety Guardrails

The examples assume that you are connected by SSH to a Prism Element Controller VM as the nutanix user. Command availability and output can vary by AOS and NCC release, hardware platform, hypervisor, and cluster state.

Before using the reference in production, apply these guardrails:

• Confirm that you are on a CVM, not an AHV host, Prism Central VM, or another appliance.
• Record the incident time, affected workloads, recent changes, and the CVM where you are connected.
• Prefer read-only commands until the failure domain is understood.
• Do not use root access for ordinary triage.
• Do not restart a service merely because it appears down once.
• Do not alter Cassandra membership, run repair operations, or remove metadata nodes without Nutanix Support.
• Do not force Curator scans or manipulate cleanup state during an active capacity or data-integrity incident.
• Check cluster fault tolerance before any action that could remove another CVM, node, disk, or failure domain.
• Preserve logs before performing an approved restart or recovery action.

The operational distinction is simple:

Start With the Symptom, Not the Service Name

The first command should narrow the failure domain. It should not change it.

The CVM Troubleshooting Evidence Flow

The most important idea in this runbook is sequence. A service can be down because it failed, because a dependency failed, because the CVM is resource constrained, because the node is isolated, or because the cluster is intentionally in a transitional state.

The diagram below shows the sequence to preserve that context.

What matters is the order. A restart performed before the protection state and logs are captured can remove the evidence needed to explain the failure.

Establish a Cluster Baseline

Run the baseline before diving into an individual service. These commands answer who, where, when, and how healthy the cluster appears at that moment.

date
hostname
ncli cluster info
svmips
cluster status
ncli alerts ls
ncli cluster get-redundancy-state
ncli cluster get-domain-fault-tolerance-status type=node

What Each Baseline Command Proves

A fast way to reduce the cluster status output to exceptions is:

cluster status | grep -v UP

Review every service row left by the filter. A clean-looking result is useful evidence, but it is not proof that the cluster is healthy. Continue with resiliency, alerts, and the service-specific evidence that matches the symptom.

Compare CVMs Instead of Trusting One Node

A local CVM can look healthy while another CVM is unstable. Use allssh to compare simple read-only evidence across the cluster.

allssh "date"
allssh "uptime"
allssh "df -h / /home"
allssh "genesis status"

Look for outliers:

• One CVM with a different time
• One CVM with a much higher load average
• One CVM with a full or nearly full filesystem
• One CVM missing a service that is present elsewhere
• One CVM showing service PIDs that change repeatedly

Large clusters can produce substantial output. Capture only what is relevant to the incident, and avoid repeatedly running expensive commands across every CVM.

Genesis Troubleshooting Reference

Genesis is the local service-management layer for the CVM. It is usually the first place to look when a service is missing, repeatedly restarting, or failing to remain stable.

Common Genesis Symptoms

• Prism reports one or more cluster services down.
• cluster status shows a service missing on one CVM.
• A service returns after a short interval and then fails again.
• The CVM booted, but the full service set did not recover.
• A post-maintenance validation shows different service states across CVMs.

Genesis Commands

genesis status
cluster status
allssh "genesis status"
watch -d genesis status

watch -d genesis status refreshes the display and highlights changes. Stop it with Ctrl+C. Stable services should normally retain stable process identifiers. A PID that repeatedly changes can indicate a crash and restart loop, although controlled maintenance and upgrades can also change PIDs.

Review the local Genesis logs next:

ls -lht /home/nutanix/data/logs/genesis*
tail -n 200 /home/nutanix/data/logs/genesis.out
grep -iE "ERROR|FATAL|exception|traceback|timeout" 
  /home/nutanix/data/logs/genesis.out | tail -n 100

Genesis Evidence and Interpretation

Genesis Escalation Point

Escalate before taking state-changing action when:

• Genesis itself is not stable.
• Multiple core services are down.
• A service repeatedly changes PID.
• FATAL logs or core-dump evidence exists.
• The cluster recently experienced an outage, abrupt power loss, upgrade failure, or node replacement.
• Fault tolerance is reduced.
• The same service failure appears on multiple CVMs.

Do not treat genesis restart, cluster start or stop operations, or a CVM reboot as routine diagnostic commands. Those actions can widen an outage and erase transient evidence.

Stargate Troubleshooting Reference

Stargate is central to the distributed storage I/O path. A Stargate issue can appear as VM latency, storage timeouts, an isolated CVM problem, a failed disk path, or a broader cluster degradation.

Common Stargate Symptoms

• VM read or write latency increases suddenly.
• Guest operating systems report storage timeouts.
• Prism reports Stargate temporarily down.
• One CVM has unusually high CPU or memory pressure.
• A node or disk event is followed by storage-service instability.
• The cluster is online, but fault tolerance is reduced.

Stargate Commands

Start with service scope:

cluster status | grep -i stargate
allssh "genesis status | grep -i stargate"

Check local CVM pressure and filesystem state:

uptime
free -m
df -h / /home
top -b -n 1 | head -n 40

Review Stargate logs around the incident time:

ls -lht /home/nutanix/data/logs/stargate*
tail -n 200 /home/nutanix/data/logs/stargate.INFO
grep -iE "ERROR|FATAL|panic|timeout|unavailable" 
  /home/nutanix/data/logs/stargate.INFO | tail -n 100

Run the targeted NCC check when unresponsive Stargate behavior is suspected:

ncc health_checks stargate_checks unresponsive_stargate_check

Recheck protection and active alerts:

ncli alerts ls
ncli cluster get-redundancy-state
ncli cluster get-domain-fault-tolerance-status type=node

Stargate Evidence and Interpretation

A single UP result is not enough. Correlate service state with workload impact, logs, and protection state.

Stargate Escalation Point

Treat the incident as urgent when:

• Data is unavailable or multiple workloads report I/O errors.
• More than one Stargate instance is affected.
• A Stargate FATAL log is created.
• Fault tolerance is zero or lower than the safe level for planned work.
• Stargate repeatedly restarts.
• Storage errors continue after the service returns.
• A disk, HBA, node, or network event is part of the same timeline.

A service restart may mask a hardware, resource, or metadata problem. Collect the service and protection evidence first.

Curator Troubleshooting Reference

Curator performs background scans and maintenance work, including activities that support cleanup and space reclamation. Curator problems often look less dramatic than a Stargate outage, but stale scans can become operationally significant when capacity is tight or a removal workflow is waiting on cleanup.

Common Curator Symptoms

• Deleted VM or snapshot capacity is not reclaimed as expected.
• Prism reports a Curator scan failure.
• A Curator job runs for an unusually long time.
• Disk removal or cleanup appears stuck.
• CVM CPU increases during scan activity.
• Capacity alerts persist after objects were deleted.

Curator Commands

Confirm service state:

cluster status | grep -i curator
allssh "genesis status | grep -i curator"

Check the command set available in the installed build, then review successful scans:

curator_cli --help
curator_cli get_last_successful_scans

Review Curator logs:

ls -lht /home/nutanix/data/logs/curator*
tail -n 200 /home/nutanix/data/logs/curator.INFO
grep -iE "ERROR|FATAL|failed|stuck|timeout" 
  /home/nutanix/data/logs/curator.INFO | tail -n 100

Correlate with alerts and local space:

ncli alerts ls
df -h / /home

Curator Evidence and Interpretation

Curator Escalation Point

Escalate when:

• Full or partial scans repeatedly fail.
• The last successful scan is outside the expected operational window and alerts persist.
• Cleanup or disk removal makes no progress.
• Capacity is critical.
• Curator logs contain repeated FATAL or timeout patterns.
• A forced scan or internal cleanup command appears to be the next step.

Do not force a scan simply to make a capacity graph move. Forced maintenance activity can add load and complicate an already constrained cluster.

Cassandra Troubleshooting Reference

Cassandra maintains distributed metadata used by the Nutanix platform. Cassandra troubleshooting has a clear safety boundary: status inspection is appropriate for operators, but ring manipulation and repair actions belong under Nutanix guidance.

Common Cassandra Symptoms

• A metadata ring or Cassandra health check fails.
• A node replacement, expansion, or reattachment does not complete.
• Cluster operations remain stuck after a node event.
• NCC or Prism reports incomplete metadata membership.
• An upgrade precheck reports Cassandra state problems.
• The cluster recovered from an outage, but metadata membership is incomplete.

Cassandra Commands

Confirm service state:

cluster status | grep -i cassandra
allssh "genesis status | grep -i cassandra"

Inspect the metadata ring:

nodetool -h 0 ring

Run the targeted NCC check:

ncc health_checks cassandra_checks cassandra_status_check

Review Cassandra logs using the filenames present in the installed release:

ls -lht /home/nutanix/data/logs/cassandra*
grep -iE "ERROR|FATAL|exception|timeout|unavailable" 
  /home/nutanix/data/logs/cassandra* | tail -n 100

What Healthy Cassandra Evidence Looks Like

Every expected CVM should appear in the ring. The key fields should indicate:

• Status: Up
• State: Normal

Anything else requires explanation. A transitional state may be legitimate during an approved cluster operation, but it should not be assumed safe during an unrelated incident.

Cassandra Escalation Point

Stop local remediation and escalate when:

• An expected CVM is absent from the ring.
• A node is not Up and Normal.
• The targeted Cassandra NCC check fails.
• A metadata operation is stuck.
• The issue follows disk replacement, node replacement, expansion, or an interrupted upgrade.
• Multiple CVMs report Cassandra errors.
• You are considering repair, removal, detach, reattach, token, or ring-membership commands.

Never use generic Apache Cassandra recovery procedures against a Nutanix cluster without Nutanix-specific guidance. The database is part of a larger distributed platform with its own orchestration and safety requirements.

NCC and Cluster Health Reference

Nutanix Cluster Check provides targeted health checks and broader validation. Use targeted checks to test a current theory, then run broader health checks when the cluster is stable enough and the incident scope justifies them.

Confirm the NCC Version

ncc --version

NCC checks evolve. A check can be added, renamed, changed, or retired across releases. Use the syntax supported by the installed NCC version and follow the KB identifier included in the output.

Targeted Service Checks

ncc health_checks system_checks cluster_services_status
ncc health_checks system_checks cvm_services_status
ncc health_checks stargate_checks unresponsive_stargate_check
ncc health_checks cassandra_checks cassandra_status_check

These checks answer different questions:

Broad Cluster Check

ncc health_checks run_all

Use a broad run after the initial symptom has been scoped, or as part of post-recovery validation. It can take time and produce findings unrelated to the active incident. Read each WARN, ERR, or FAIL in context, and use the associated Nutanix KB guidance rather than treating every non-PASS line as the same severity.

Symptom-Based Runbook

Prism Is Unavailable but CVM SSH Works

Run:

date
cluster status
genesis status
ncli alerts ls
allssh "genesis status"

Then determine whether the issue is local to one CVM or present across the cluster. Check Genesis logs for repeated startup failures. Escalate if multiple services are missing, PIDs are unstable, or the cluster has no stable management path.

VM Storage Latency or I/O Errors

Run:

cluster status | grep -i stargate
ncc health_checks stargate_checks unresponsive_stargate_check
ncli alerts ls
ncli cluster get-redundancy-state
ncli cluster get-domain-fault-tolerance-status type=node

Then inspect Stargate logs and local CVM resource pressure. Escalate immediately for data unavailability, multiple affected Stargate instances, FATAL logs, or reduced protection.

Capacity Is Not Reclaimed

Run:

cluster status | grep -i curator
curator_cli get_last_successful_scans
ncli alerts ls
df -h / /home

Then review Curator logs for failed, stuck, or timeout patterns. Escalate when scans repeatedly fail, capacity is critical, or cleanup and removal workflows stop progressing.

A Node Replacement or Expansion Leaves Metadata Warnings

Run:

cluster status
nodetool -h 0 ring
ncc health_checks cassandra_checks cassandra_status_check
ncli cluster get-domain-fault-tolerance-status type=node

Do not proceed with additional topology changes unless the expected CVMs are present and normal. Escalate any unexpected ring state.

Services Keep Restarting

Run:

watch -d genesis status

In a second session, review the relevant service log and Genesis log. Record the service name, PID changes, first failure time, and frequency. Escalate before restarting the service again.

Post-Outage or Post-Maintenance Validation

Run:

cluster status | grep -v UP
nodetool -h 0 ring
ncli cluster get-redundancy-state
ncli cluster get-domain-fault-tolerance-status type=node
ncc health_checks system_checks cluster_services_status
ncc health_checks system_checks cvm_services_status

Then validate Prism, active alerts, and representative workload access. A cluster is not recovered merely because Prism opens.

Evidence Collection Before Escalation

A useful support case contains a timeline and a compact evidence set, not a random collection of screenshots.

Collect:

• Cluster name and AOS version
• NCC version
• Incident start time and timezone
• Affected VMs, nodes, disks, or services
• Recent upgrades, maintenance, power, network, or hardware changes
• cluster status
• Relevant genesis status output
• Redundancy and node fault-tolerance output
• Cassandra ring output when metadata is involved
• Relevant NCC result and KB identifier
• Service INFO and FATAL excerpts around the incident
• Whether the issue is local, multi-node, or cluster-wide

Collect a Logbay Bundle

Check CVM space first:

df -h /home
logbay collect --help

A basic collection uses the default time window supported by the installed Logbay release:

logbay collect --aggregate=true

Bundles are normally written under:

/home/nutanix/data/logbay/bundles

When the incident occurred outside the default collection window, use the installed command help and the applicable Nutanix documentation to select an exact start time and duration. On older supported releases where Logbay is unavailable, the release documentation may direct you to the legacy NCC log collector instead.

Do not delete arbitrary files from /home/nutanix/data/logs to make room for a bundle. If CVM space is constrained, follow the Nutanix procedure for that release or open a support case.

Actions That Should Not Be Part of Initial Triage

The following actions can be valid in a controlled recovery plan, but they do not belong in the first evidence-gathering pass:

• Restarting Genesis or individual cluster services
• Running cluster start or stop operations
• Powering off or rebooting a CVM
• Entering or leaving CVM maintenance mode
• Running Cassandra repair or ring-membership commands
• Removing or reattaching metadata nodes
• Forcing Curator scans or internal cleanup work
• Deleting CVM logs, bundles, or data files manually
• Continuing node, disk, or rack maintenance while fault tolerance is reduced
• Applying a command copied from a different AOS release without checking local help and current documentation

A good operational rule is to stop when the next command would change state rather than collect evidence.

Validation After Recovery

Recovery validation should repeat the same evidence chain used during triage.

cluster status | grep -v UP
nodetool -h 0 ring
ncli cluster get-redundancy-state
ncli cluster get-domain-fault-tolerance-status type=node
ncc health_checks system_checks cluster_services_status
ncc health_checks system_checks cvm_services_status

Then verify:

• Service PIDs remain stable.
• Active alerts are resolved or understood.
• Every expected Cassandra node is Up and Normal.
• Node-level fault tolerance has returned to the expected state.
• Prism is responsive.
• Representative VMs can read and write normally.
• Storage latency has returned to baseline.
• Curator has resumed successful scans when cleanup was involved.
• No new FATAL logs appeared after recovery.
• The incident timeline and recovery action are documented.

Compact Command Reference by Service

Conclusion

The most useful Nutanix CVM command is not necessarily the most advanced one. It is the command that narrows the failure domain without changing the system before the evidence is understood.

For most incidents, the reliable sequence is consistent: establish the cluster baseline, compare CVMs, verify protection state, inspect the service that owns the symptom, run a targeted NCC check, preserve logs, and escalate before changing cluster state.

Stargate, Curator, Cassandra, and Genesis each fail differently, but operators should approach them with the same discipline. A status result must be correlated with logs. A service restart must be distinguished from a stable recovery. A healthy Prism page must be backed by fault tolerance, ring membership, and workload validation.

That discipline turns a command list into an operational runbook and reduces the chance that troubleshooting becomes the next outage.

External References

• Nutanix Support & Insights: General troubleshooting knowledge base article
  Canonical URL: https://portal.nutanix.com/kb/11619
• Nutanix Support & Insights: Verifying cluster recovery after a power outage
  Canonical URL: https://portal.nutanix.com/kb/12328
• Nutanix Support & Insights: NCC Health Check: cluster_services_status
  Canonical URL: https://portal.nutanix.com/kb/3378
• Nutanix Support & Insights: NCC Health Check: cvm_services_status
  Canonical URL: https://portal.nutanix.com/kb/2472
• Nutanix Support & Insights: NCC Health Check: unresponsive_stargate_check
  Canonical URL: https://portal.nutanix.com/kb/3527
• Nutanix Support & Insights: NCC Health Check: cassandra_status_check
  Canonical URL: https://portal.nutanix.com/kb/1547
• Nutanix Documentation: AOS 7.5 – cluster
  Canonical URL: https://portal.nutanix.com/docs/Command-Ref-AOS-v7_5%3Aacl-ncli-cluster-auto-r.html
• Nutanix Documentation: NCC 5.2.0 – Logbay Log Collection (Command Line)
  Canonical URL: https://portal.nutanix.com/docs/NCC-Guide-NCC-v5_2%3Ancc-ncc-logbay-cli-r.html
• Nutanix Support & Insights: Alert – A1030 – StargateTemporarilyDown
  Canonical URL: https://portal.nutanix.com/kb/3784
• Nutanix Support & Insights: Alert – A1081 – CuratorScanFailure
  Canonical URL: https://portal.nutanix.com/kb/3786
• Nutanix Cloud Bible: Controller VM (CVM)
  Canonical URL: https://www.nutanixbible.com/2c-book-of-basics-hyperconverged-platform.html
• Nutanix Community: Disk Space Usage High alert in Prism
  Canonical URL: https://next.nutanix.com/intelligent-operations-26/disk-space-usage-high-alert-in-prism-38199

The post Nutanix CVM Troubleshooting Command Reference: Stargate, Curator, Cassandra, Genesis, and Cluster Health appeared first on Digital Thought Disruption.