NSX 4.x Health Check Handbook

1. Overview & Purpose

This handbook provides a complete, step-by-step health check procedure for VMware NSX 4.x deployed within a VCF 9.0 environment. It is designed for VMware administrators who need to verify NSX health during:

• Routine maintenance windows — Weekly or monthly proactive checks
• Pre/post-upgrade validation — Before and after NSX patches or upgrades
• Incident troubleshooting — When connectivity, firewall, or routing issues occur
• Environment handover — Documenting health state for audits or transfers

What This Document Covers

Health Check Methodology

Each check in this handbook follows a consistent format:

1. What to check — Description of the component and why it matters
2. How to check — Exact CLI command or API call (copy-paste ready)
3. Expected output — What a healthy result looks like
4. Pass / Warn / Fail criteria — Clear thresholds with visual indicators
5. Remediation — What to do if the check fails

2. Prerequisites

Required Access

Required Tools

Environment Variables Setup

# NSX Manager
export NSX_VIP="nsx-vip.lab.local"
export NSX_NODE1="nsx-mgr-01.lab.local"
export NSX_NODE2="nsx-mgr-02.lab.local"
export NSX_NODE3="nsx-mgr-03.lab.local"
export NSX_USER="admin"
export NSX_PASS="YourPassword123!"

# Convenience function for NSX API calls
nsx_api() {
  curl -sk -u "$NSX_USER:$NSX_PASS" \
    -H "Content-Type: application/json" \
    "https://$NSX_VIP$1" 2>/dev/null
}

# Example usage:
# nsx_api /api/v1/cluster/status | jq .

3. Quick Reference — All Checks Summary

4. NSX Manager Cluster Health

4.1 Cluster Status (API + CLI)

What: Verify the NSX Manager cluster is stable with all 3 nodes participating.

Why: A degraded management cluster means reduced redundancy. An unstable cluster can cause configuration failures and data loss.

API Method

# Get cluster status
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/cluster/status" | jq .

Expected Output (Healthy):

{
  "detailed_cluster_status": {
    "overall_status": "STABLE",
    "groups": [
      {
        "group_type": "CONTROLLER",
        "group_status": "STABLE",
        "members": [
          {
            "member_uuid": "abc123...",
            "member_status": "UP",
            "member_fqdn": "nsx-mgr-01.lab.local"
          },
          {
            "member_uuid": "def456...",
            "member_status": "UP",
            "member_fqdn": "nsx-mgr-02.lab.local"
          },
          {
            "member_uuid": "ghi789...",
            "member_status": "UP",
            "member_fqdn": "nsx-mgr-03.lab.local"
          }
        ]
      },
      {
        "group_type": "MANAGER",
        "group_status": "STABLE",
        "members": [...]
      },
      {
        "group_type": "POLICY",
        "group_status": "STABLE",
        "members": [...]
      }
    ]
  },
  "mgmt_cluster_status": {
    "status": "STABLE"
  },
  "control_cluster_status": {
    "status": "STABLE"
  }
}

CLI Method (SSH to any NSX Manager)

ssh admin@$NSX_NODE1
# Then run:
get cluster status

Expected Output:

Cluster Id: a1b2c3d4-e5f6-7890-abcd-ef1234567890
Overall Status: STABLE

Group Type: CONTROLLER
  Group Status: STABLE
  Leader: nsx-mgr-01.lab.local

Group Type: MANAGER
  Group Status: STABLE
  Leader: nsx-mgr-01.lab.local

Group Type: POLICY
  Group Status: STABLE
  Leader: nsx-mgr-02.lab.local

Pass / Warn / Fail

4.2 Individual Node Health

What: Check the health status of each individual NSX Manager node.

Why: A node can be in the cluster but unhealthy (high load, service failures).

API Method

# List all cluster nodes with health
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/cluster/nodes" | jq '.results[] | {
    fqdn: .fqdn,
    ip: .ip_address,
    status: .status,
    version: .node_version
  }'

Check Individual Node Appliance Health

# Per-node health via appliance API
for NODE in $NSX_NODE1 $NSX_NODE2 $NSX_NODE3; do
  echo "=== $NODE ==="
  curl -sk -u "$NSX_USER:$NSX_PASS" \
    "https://$NODE/api/v1/node" | jq '{
      hostname: .hostname,
      kernel_version: .kernel_version,
      product_version: .product_version,
      uptime: .system_time
    }'
done

Pass / Warn / Fail

4.3 Service Status Verification

What: Verify all critical NSX Manager services are running on each node.

Why: Individual service failures can cause specific feature outages even when the cluster appears healthy.

CLI Method (SSH to each NSX Manager)

ssh admin@$NSX_NODE1
get services

Expected Output (all running):

Service Name          Service Status
-------------------------------------
async_replicator      running
cluster-manager       running
controller            running
corfu                 running
corfu-nonconfig       running
datastore             running
http                  running
install-upgrade       running
liagent               running
manager               running
migration-coordinator running
monitoring            running
nsx-message-bus       running
nsx-ui-plugin         running
phonehome             running
platform-client       running
policy                running
proton                running
search                running
syslog-server         running
upgrade-coordinator   running

API Method

# Get node services status
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/node/services" | jq '.results[] | {
    service: .service_name,
    status: .service_status.runtime_state
  }'

Critical Services (must be running)

5. NSX Manager Resource Utilization

5.1 CPU & Memory

What: Check CPU and memory utilization on each NSX Manager node.

Why: NSX Manager nodes under resource pressure may respond slowly or fail to process API requests, causing cascading issues across the environment.

CLI Method

ssh admin@$NSX_NODE1
# CPU utilization
get node-stats

API Method

# Node status with resource utilization
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_NODE1/api/v1/node/status" | jq '{
    cpu_cores: .cpu_cores,
    load_average: .load_average,
    mem_total: .mem_total,
    mem_used: .mem_used,
    mem_cache: .mem_cache,
    swap_total: .swap_total,
    swap_used: .swap_used,
    uptime: .uptime
  }'

Expected Output (Healthy):

{
  "cpu_cores": 12,
  "load_average": [1.2, 1.5, 1.8],
  "mem_total": 49152,
  "mem_used": 32768,
  "mem_cache": 8192,
  "swap_total": 8192,
  "swap_used": 0,
  "uptime": 8640000
}

Pass / Warn / Fail

5.2 Disk & Filesystem

What: Check disk space utilization on all NSX Manager filesystem partitions.

Why: Full disks cause service crashes, log loss, and database corruption.

CLI Method

ssh admin@$NSX_NODE1
# As admin:
get filesystem-info

API Method

curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_NODE1/api/v1/node/status" | jq '.filesystem'

Root SSH Method (more detail)

ssh root@$NSX_NODE1
df -h

Expected Output:

Filesystem      Size  Used  Avail  Use%  Mounted on
/dev/sda2        40G   15G    23G   40%  /
/dev/sda3        80G   28G    48G   37%  /nonconfig-datastore
/dev/sda5        40G   12G    26G   32%  /config-datastore
/dev/sda6        20G    4G    15G   21%  /image
tmpfs            24G   12M    24G    1%  /dev/shm

Critical Filesystem Thresholds

5.3 Thresholds & Alarms

What: Check if any resource-related alarms are active on NSX Manager nodes.

API Method

# Check system alarms related to resources
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/alarms?feature_name=system" | jq '.results[] | {
    id: .id,
    severity: .severity,
    description: .description,
    status: .status,
    entity_id: .entity_id
  }'

6. Certificate Health

6.1 List All Certificates

What: Retrieve all certificates managed by NSX and verify none are expired or expiring soon.

Why: Expired certificates cause communication failures between NSX components, transport nodes, and external integrations.

API Method

# List all certificates
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/trust-management/certificates" | jq '.results[] | {
    id: .id,
    display_name: .display_name,
    not_before: .not_before,
    not_after: .not_after,
    subject: .pem_encoded | split("\n")[0]
  }'

Extract Expiry Dates for Quick Review

# Check certificate expiry with days remaining
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/trust-management/certificates" | jq -r '
  .results[] |
  "\(.display_name)\t\(.not_after)\t\(
    ((.not_after / 1000) - (now | floor)) / 86400 | floor
  ) days remaining"'

Expected Output:

nsx-mgr-01-cert     1774003200000    365 days remaining
nsx-mgr-02-cert     1774003200000    365 days remaining
nsx-mgr-03-cert     1774003200000    365 days remaining
nsx-api-cert        1774003200000    365 days remaining
mp-cluster-cert     1774003200000    365 days remaining

6.2 Expiry Verification

Pass / Warn / Fail

Verify Certificate via OpenSSL

# Verify the NSX Manager certificate externally
echo | openssl s_client -connect $NSX_VIP:443 2>/dev/null | \
  openssl x509 -noout -dates -subject -issuer

Expected Output:

notBefore=Jan 15 00:00:00 2026 GMT
notAfter=Jan 15 00:00:00 2028 GMT
subject=CN = nsx-vip.lab.local
issuer=CN = NSX CA

7. Backup Verification

7.1 Backup Configuration

What: Verify that NSX Manager backups are configured and running on schedule.

Why: Without valid backups, any cluster failure could result in complete NSX configuration loss.

API Method

# Get backup configuration
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/cluster/backups/config" | jq '{
    backup_enabled: .backup_enabled,
    backup_schedule: .backup_schedule,
    remote_file_server: .remote_file_server.server,
    remote_path: .remote_file_server.directory_path
  }'

Expected Output (Healthy):

{
  "backup_enabled": true,
  "backup_schedule": {
    "resource_type": "WeeklyBackupSchedule",
    "days_of_week": ["MONDAY", "WEDNESDAY", "FRIDAY"],
    "hour_of_day": 2,
    "minute_of_day": 0
  },
  "remote_file_server": "backup-server.lab.local",
  "remote_path": "/backups/nsx/"
}

7.2 Backup History

# Check backup history
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/cluster/backups/history" | jq '.results[0:5] | .[] | {
    start_time: .start_time,
    end_time: .end_time,
    backup_status: .success,
    node: .node_id
  }'

Pass / Warn / Fail

8. Transport Node Health

8.1 Transport Node Status

What: Verify all ESXi hosts prepared as NSX transport nodes are in a healthy state.

Why: Transport node failures mean VMs on that host lose NSX networking (overlay, DFW, load balancing).

API Method

# Get all transport node states
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/transport-nodes/state" | jq '.results[] | {
    transport_node_id: .transport_node_id,
    node_deployment_state: .node_deployment_state.state,
    state: .state,
    status: .status,
    host_node_deployment_status: .host_node_deployment_status
  }'

Expected Output (Healthy):

{
  "transport_node_id": "abc123",
  "node_deployment_state": "NODE_READY",
  "state": "success",
  "status": "UP"
}

Summary View

# Compact summary of all transport node states
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/transport-nodes/state?status=UP" | jq '.result_count'

curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/transport-nodes/state?status=DOWN" | jq '.result_count'

curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/transport-nodes/state?status=DEGRADED" | jq '.result_count'

Pass / Warn / Fail

8.2 ESXi nsxcli Commands

What: Verify NSX components on individual ESXi hosts via the nsxcli shell.

SSH to ESXi and Enter nsxcli

ssh root@<esxi-host>
nsxcli

Check Manager Connection

> get managers

Expected Output:

Manager IP     : 192.168.1.71
Manager IP     : 192.168.1.72
Manager IP     : 192.168.1.73
Connected to   : 192.168.1.71

Check Controller Connection

> get controllers

Expected Output:

Controller IP  : 192.168.1.71
Controller IP  : 192.168.1.72
Controller IP  : 192.168.1.73
Connected to   : 192.168.1.72
Status         : Connected

8.3 NSX Agent Status

What: Verify the NSX agent processes (nsx-mpa, nsx-proxy) are running on each ESXi host.

Why: These agents handle management plane communication and data plane programming.

CLI Method (on ESXi host)

# Check NSX services on ESXi
/etc/init.d/nsx-mpa status
/etc/init.d/nsx-proxy status
/etc/init.d/nsx-context-mux status

Expected Output:

nsx-mpa is running
nsx-proxy is running
nsx-context-mux is running

Alternative Check

esxcli software vib list | grep -i nsx

Expected Output (shows installed NSX VIBs):

nsx-esx-datapath       4.2.0.0.0-12345678   VMW  VMwareCertified  2026-01-15
nsx-mpa                4.2.0.0.0-12345678   VMW  VMwareCertified  2026-01-15
nsx-platform-client    4.2.0.0.0-12345678   VMW  VMwareCertified  2026-01-15
nsx-proxy              4.2.0.0.0-12345678   VMW  VMwareCertified  2026-01-15

8.4 Port Connectivity

What: Verify ESXi transport nodes can reach NSX Managers on required ports.

Port Requirements

Connectivity Test from ESXi

# Test connectivity to NSX Manager ports from ESXi
nc -zv $NSX_VIP 1234
nc -zv $NSX_VIP 1235
nc -zv $NSX_VIP 5671
nc -zv $NSX_VIP 443

Expected Output:

Connection to nsx-vip.lab.local 1234 port [tcp/*] succeeded!
Connection to nsx-vip.lab.local 1235 port [tcp/*] succeeded!
Connection to nsx-vip.lab.local 5671 port [tcp/*] succeeded!
Connection to nsx-vip.lab.local 443 port [tcp/https] succeeded!

9. TEP Connectivity

9.1 vmkping Tests with MTU Validation

What: Verify Tunnel Endpoint (TEP) connectivity between all ESXi hosts with proper MTU.

Why: TEP connectivity is the foundation of NSX overlay networking. Failed TEP = no overlay connectivity between VMs on different hosts.

Standard vmkping (Default MTU)

# From ESXi host, ping another host's TEP IP
vmkping -I vmk10 -d -s 1572 <remote-TEP-IP>

Test All TEP Pairs

# Script to test TEP connectivity from current host
REMOTE_TEPS="192.168.12.75 192.168.12.76 192.168.12.82"
for TEP in $REMOTE_TEPS; do
  echo "Testing TEP: $TEP"
  vmkping -I vmk10 -d -s 1572 -c 3 $TEP
  echo "---"
done

Expected Output (Healthy):

PING 192.168.12.75 (192.168.12.75): 1572 data bytes
1580 bytes from 192.168.12.75: icmp_seq=0 ttl=64 time=0.543 ms
1580 bytes from 192.168.12.75: icmp_seq=1 ttl=64 time=0.421 ms
1580 bytes from 192.168.12.75: icmp_seq=2 ttl=64 time=0.389 ms

--- 192.168.12.75 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss

Pass / Warn / Fail

9.2 BFD Session Verification

What: Verify Bidirectional Forwarding Detection (BFD) sessions between transport nodes.

Why: BFD provides fast failure detection for TEP tunnels. Down BFD = NSX considers the tunnel failed.

CLI Method (nsxcli on ESXi)

ssh root@<esxi-host>
nsxcli
> get bfd-sessions

Expected Output:

BFD Session          Remote IP        State   Diagnostic
-------------------------------------------------------
192.168.12.75        192.168.12.75    UP      No Diagnostic
192.168.12.76        192.168.12.76    UP      No Diagnostic
192.168.12.82        192.168.12.82    UP      No Diagnostic

Pass / Warn / Fail

9.3 Tunnel Status

What: Verify overlay tunnel status between transport nodes.

nsxcli Method

> get logical-switches

Expected Output:

Logical Switch UUID                          VNI     Port Count
---------------------------------------------------------------
abc123-def456-789012-abcdef-123456789012    71001   5
def456-789012-abcdef-123456-789012abcdef    71002   3

> get tunnel-ports

Expected Output:

Tunnel Port  Remote IP        Encap  Status
--------------------------------------------
vxlan_12     192.168.12.75    Geneve UP
vxlan_13     192.168.12.76    Geneve UP
vxlan_14     192.168.12.82    Geneve UP

10. Edge Node Health

10.1 Edge Cluster Status

What: Verify the Edge cluster and all member Edge nodes are operational.

Why: Edge nodes provide north-south routing, NAT, load balancing, VPN, and gateway firewall services. Edge failure = services outage.

API Method

# List edge clusters
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/edge-clusters" | jq '.results[] | {
    id: .id,
    display_name: .display_name,
    member_count: (.members | length),
    deployment_type: .deployment_type
  }'

# Get edge cluster status
EDGE_CLUSTER_ID="<edge-cluster-id>"
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/edge-clusters/$EDGE_CLUSTER_ID/status" | jq .

Expected Output:

{
  "edge_cluster_id": "abc123...",
  "member_status": [
    {
      "member_index": 0,
      "transport_node_id": "tn-edge-01",
      "status": "UP"
    },
    {
      "member_index": 1,
      "transport_node_id": "tn-edge-02",
      "status": "UP"
    }
  ]
}

10.2 HA State Verification

What: Verify Edge High Availability is correctly configured with proper ACTIVE/STANDBY states.

API Method — Tier-0 HA Status

# Get Tier-0 gateways
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/tier-0s" | jq '.results[] | {
    id: .id,
    display_name: .display_name,
    ha_mode: .ha_mode,
    failover_mode: .failover_mode
  }'

CLI Method (SSH to Edge Node)

ssh admin@<edge-node>
get high-availability status

Expected Output:

Active-Standby Status
  Service Router    : ACTIVE
  Distributed Router: ACTIVE
  HA Channel State  : UP
  Peer Status       : STANDBY (edge-02)

Pass / Warn / Fail

10.3 Service Channels

What: Verify datapath and routing services on edge nodes.

CLI Method (on Edge Node)

ssh admin@<edge-node>
get service status

Expected Output:

Service          Status
--------------------------
dataplane        running
edge-health-agg  running
nestdb           running
nsx-edge-mpa     running
nsx-proxy        running
syslog           running

get interfaces

Expected Output:

Interface    IP Address         MAC               MTU   Admin Status
--------------------------------------------------------------------
fp-eth0      192.168.10.1/24    00:50:56:xx:xx:01 1500  UP
fp-eth1      192.168.20.1/24    00:50:56:xx:xx:02 1500  UP
fp-eth2      169.254.0.1/28     02:50:56:xx:xx:03 1500  UP

11. Routing & BGP Health

11.1 BGP Neighbor Summary

What: Verify BGP sessions with upstream physical routers are established and routes are being exchanged.

Why: BGP peering is critical for north-south traffic flow. Failed BGP = no external connectivity.

CLI Method (on Edge Node)

ssh admin@<edge-node>
get bgp neighbor summary

Expected Output:

BGP Summary:
  Router ID: 192.168.10.1
  Local AS:  65001

Neighbor        AS      State        Up/Down   Prefixes Received
-----------------------------------------------------------------
192.168.10.254  65000   Established  5d12h     12
192.168.20.254  65000   Established  5d12h     12

API Method

# Get BGP neighbors via policy API
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/tier-0s/<tier0-id>/locale-services/<locale-service-id>/bgp/neighbors/status" | jq .

Pass / Warn / Fail

11.2 Route Table Verification

What: Verify the routing table contains expected routes for all network segments.

CLI Method (Edge Node)

get route-table

Expected Output (abbreviated):

Flags: c - connected, s - static, b - BGP, ns - nsx_static
  > - selected route, * - FIB route

b>*  0.0.0.0/0         via 192.168.10.254    fp-eth0   [20/0]
c>*  192.168.10.0/24   directly connected    fp-eth0
c>*  192.168.20.0/24   directly connected    fp-eth1
b>*  10.0.0.0/8        via 192.168.10.254    fp-eth0   [20/0]
ns>* 172.16.0.0/16     via 169.254.0.2       linked    [3/0]
ns>* 172.17.0.0/16     via 169.254.0.2       linked    [3/0]

Verify Expected Routes

11.3 Gateway Realized State

What: Verify Tier-0 and Tier-1 gateways are fully realized (configured intent matches actual state).

API Method — Tier-0

# Check Tier-0 realized state
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/tier-0s/<tier0-id>/state" | jq '{
    state: .state,
    details: .details
  }'

API Method — Tier-1

# Check Tier-1 realized state
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/tier-1s" | jq '.results[] | {
    display_name: .display_name,
    id: .id
  }'

# Then for each Tier-1:
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/tier-1s/<tier1-id>/state" | jq '{
    state: .state,
    details: .details
  }'

Pass / Warn / Fail

12. Distributed Firewall Health

12.1 DFW Rule Realization Status

What: Verify all DFW security policies and rules have been successfully realized (pushed) to transport nodes.

Why: Unrealized rules mean security policies are not being enforced, creating security gaps.

API Method

# Check DFW realization status
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/realized-state/status?intent_path=/infra/domains/default/security-policies" | jq '{
    publish_status: .publish_status,
    consolidated_status: .consolidated_status.consolidated_status
  }'

Detailed Status Per Policy

# List all security policies and their realization state
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/domains/default/security-policies" | jq '.results[] | {
    display_name: .display_name,
    id: .id,
    rule_count: (.rules | length)
  }'

# Check realization for a specific policy
POLICY_ID="<policy-id>"
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/realized-state/status?intent_path=/infra/domains/default/security-policies/$POLICY_ID" | jq .

Pass / Warn / Fail

12.2 ESXi dvfilter Verification

What: Verify DFW filters are attached to VM vNICs on ESXi hosts.

Why: If dvfilter is not attached, DFW rules are not applied to that VM regardless of realization status.

CLI Method (on ESXi host)

# List all dvfilters
summarize-dvfilter

Expected Output (abbreviated):

world 12345678 vmm0:MyVM vcUuid:'50 12 ab cd ...'
 ++  port 50331651 myvm.eth0
    vNic slot 2
    name: nic-12345678-eth0-vmware-sfw.2
    agentName: vmware-sfw
    state: IOChain Attached

12.3 vsipioctl Commands

What: Verify DFW rules are programmed on the ESXi datapath using vsipioctl.

Get Filter List

vsipioctl getfilters

Expected Output:

Filter Name                              : nic-12345678-eth0-vmware-sfw.2
Filter Rules:
  Rule Count  : 47
  Applied To  : myvm.eth0

View Rules for a Specific Filter

vsipioctl getrules -f nic-12345678-eth0-vmware-sfw.2

Expected Output (abbreviated):

ruleset domain-c8:1001 {
  # DFW Section: Application Rules
  rule 1001 at 1 inout protocol any from any to any accept with log;
  rule 1002 at 2 inout protocol tcp from addrset src-001 to addrset dst-001 port 443 accept;
  rule 1003 at 3 inout protocol any from any to any drop;
}

View Address Sets

vsipioctl getaddrsets -f nic-12345678-eth0-vmware-sfw.2

13. Gateway Firewall Health

What: Verify Gateway Firewall rules on Tier-0 and Tier-1 gateways are realized and synchronized between HA edge pairs.

Why: Gateway firewall protects north-south traffic. Desynchronized rules between active/standby edges create intermittent security gaps during failover.

API Method — Gateway Firewall Status

# Tier-0 Gateway Firewall policies
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/tier-0s/<tier0-id>/gateway-firewall" | jq .

# Tier-1 Gateway Firewall policies
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/tier-1s/<tier1-id>/gateway-firewall" | jq .

Check Rule Realization on Edge

# SSH to edge node
ssh admin@<edge-node>
get firewall rules

Expected Output:

Rule ID    Action   Direction  Source          Destination     Service    Logged
---------------------------------------------------------------------------------
1001       allow    in-out     10.0.0.0/8      any            HTTPS       yes
1002       allow    in-out     any             10.0.0.0/8     SSH         yes
2001       drop     in-out     any             any            any         yes

Verify Sync Between Active/Standby

# On ACTIVE edge:
get firewall rules | wc -l

# On STANDBY edge:
get firewall rules | wc -l

The rule count should be identical on both edges.

Pass / Warn / Fail

14. Segments & Logical Switches

14.1 Segment Status

What: Verify all NSX segments are operational and correctly realized.

API Method

# List all segments with status
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/segments" | jq '.results[] | {
    display_name: .display_name,
    id: .id,
    type: .type,
    vlan_ids: .vlan_ids,
    transport_zone_path: .transport_zone_path,
    admin_state: .admin_state
  }'

Check Segment Realized State

# For each segment, check realization
SEGMENT_ID="<segment-id>"
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/policy/api/v1/infra/segments/$SEGMENT_ID/state" | jq '{
    state: .state,
    details: .details
  }'

Pass / Warn / Fail

14.2 MAC / ARP / VTEP Tables

What: Verify logical switch forwarding tables are populated correctly.

nsxcli on ESXi

nsxcli
> get logical-switch <VNI> mac-table

Expected Output:

Inner MAC          Outer MAC          Outer IP         Flags
--------------------------------------------------------------
00:50:56:aa:bb:01  00:50:56:cc:dd:01  192.168.12.75   L
00:50:56:aa:bb:02  00:50:56:cc:dd:02  192.168.12.76   R

• L = Local (VM is on this host)
• R = Remote (VM is on another host, reachable via VTEP)

> get logical-switch <VNI> arp-table

Expected Output:

IP Address       MAC Address        Flags
-------------------------------------------
172.16.1.10      00:50:56:aa:bb:01  L
172.16.1.11      00:50:56:aa:bb:02  R

> get logical-switch <VNI> vtep-table

Expected Output:

VTEP IP           VTEP MAC             Segment ID  Is Local
------------------------------------------------------------
192.168.12.74     00:50:56:66:xx:01    192.168.12.0  true
192.168.12.75     00:50:56:66:xx:02    192.168.12.0  false
192.168.12.76     00:50:56:66:xx:03    192.168.12.0  false

15. NSX Alarms

What: Check for active alarms on the NSX Manager that indicate current or impending issues.

Why: NSX generates alarms for certificate expiry, resource exhaustion, connectivity failures, and more. Unresolved alarms indicate health issues.

API Method — All Active Alarms

# Get all open alarms
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/alarms?status=OPEN,ACKNOWLEDGED&sort_by=severity" | jq '.results[] | {
    alarm_id: .id,
    severity: .severity,
    feature: .feature_name,
    event_type: .event_type,
    description: .description,
    entity: .entity_id,
    last_reported: .last_reported_time,
    status: .status
  }'

Filter Critical Alarms Only

curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/alarms?status=OPEN&severity=CRITICAL" | jq '.result_count'

Key Alarm Categories

Pass / Warn / Fail

Acknowledge / Resolve Alarms via API

# Acknowledge an alarm
curl -sk -X POST -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/alarms/<alarm-id>?action=set_status&new_status=ACKNOWLEDGED"

# Resolve an alarm (after fixing the root cause)
curl -sk -X POST -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/alarms/<alarm-id>?action=set_status&new_status=RESOLVED"

16. Port Reference Table

NSX Manager Ports

Transport Node (ESXi) Ports

Edge Node Ports

17. Common Issues & Remediation

17.1 Control Plane Connectivity Issues

17.2 Transport Node Preparation Failures

17.3 Certificate Expiry Impacts

17.4 Cluster Split-Brain

Symptoms:

• Two NSX Manager groups each claiming to be leader
• Conflicting configuration states
• Intermittent API responses with different data

Resolution:

1. Identify the minority partition (fewer nodes)
2. Power off minority nodes
3. Verify majority cluster stabilizes
4. Power on minority nodes one at a time
5. Monitor cluster rejoin via get cluster status

18. CLI Quick Reference Card

NSX Manager CLI (SSH as admin)

ESXi nsxcli Commands

ESXi Host Commands (as root)

Edge Node CLI

19. API Quick Reference

Authentication

# Basic authentication (used throughout this document)
curl -sk -u "admin:password" "https://$NSX_VIP/api/v1/..."

# Session-based authentication
curl -sk -c cookies.txt -X POST \
  -d '{"j_username":"admin","j_password":"password"}' \
  "https://$NSX_VIP/api/session/create"

# Subsequent calls with session cookie
curl -sk -b cookies.txt "https://$NSX_VIP/api/v1/..."

Key API Endpoints

Common API Query Parameters

Version Check

# Get NSX Manager version
curl -sk -u "$NSX_USER:$NSX_PASS" \
  "https://$NSX_VIP/api/v1/node/version" | jq .

Expected Output:

{
  "node_version": "4.2.0.0.0.12345678",
  "product_version": "4.2.0.0.0.12345678"
}