vCenter Server Health Check Handbook

1. Overview & Purpose

This handbook provides a complete, step-by-step health check procedure for VMware vCenter Server 8.x deployed within a VCF 9.0 environment. It is designed for VMware administrators who need to verify vCenter Server health during:

• Routine maintenance windows -- Weekly or monthly proactive checks
• Pre/post-upgrade validation -- Before and after vCenter patches or upgrades
• Incident troubleshooting -- When service, connectivity, or performance issues occur
• Environment handover -- Documenting health state for audits or transfers
• VCF lifecycle operations -- Before and after SDDC Manager orchestrated updates

What This Document Covers

Health Check Methodology

Each check in this handbook follows a consistent format:

1. What to check -- Description of the component and why it matters
2. How to check -- Exact CLI command or API call (copy-paste ready)
3. Expected output -- What a healthy result looks like
4. Pass / Warn / Fail criteria -- Clear thresholds with visual indicators
5. Remediation -- What to do if the check fails

2. Prerequisites

Required Access

Required Tools

Environment Setup

Set these variables before running commands:

# vCenter connection variables
export VC_FQDN="vcenter01.lab.local"
export VC_USER="administrator@vsphere.local"
export VC_PASS='YourPasswordHere'

# Obtain a session token
export VC_TOKEN=$(curl -sk -X POST \
  "https://${VC_FQDN}/api/session" \
  -u "${VC_USER}:${VC_PASS}" | tr -d '"')

# Verify the token was obtained
echo "Session Token: ${VC_TOKEN}"

Enable SSH on vCenter Appliance

If SSH is not enabled, enable it via VAMI:

1. Navigate to https://$VC_FQDN:5480
2. Log in with root credentials
3. Go to Access > SSH Login > Edit > Enable
4. Alternatively via API:

curl -sk -X PUT \
  "https://${VC_FQDN}/api/appliance/access/ssh" \
  -H "vmware-api-session-id: ${VC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

3. Quick Reference -- All Checks Summary

4. VPXD Service Health

The VPXD (VMware VirtualCenter Server Daemon) is the core service of vCenter Server. It manages ESXi hosts, virtual machines, storage, and networking. If VPXD is down, the entire vCenter is non-functional.

4.1 VPXD Process Status

CLI Check (SSH to vCenter Appliance)

# Check VPXD service status
service-control --status vmware-vpxd

Expected Output (Healthy):

VMware vCenter Server:Status: RUNNING

Alternative: systemctl check

systemctl status vmware-vpxd

Expected Output (Healthy):

● vmware-vpxd.service - VMware vCenter Server
   Loaded: loaded (/usr/lib/systemd/system/vmware-vpxd.service; enabled)
   Active: active (running) since Mon 2026-03-23 10:15:22 UTC; 3 days ago
 Main PID: 5432 (vpxd)
    Tasks: 312
   Memory: 2.1G
   CGroup: /system.slice/vmware-vpxd.service
           └─5432 /usr/lib/vmware-vpxd/vpxd

API Check

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/services/vmware-vpxd" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq .

4.2 VPXD Restart Procedures

Graceful Restart

# Stop then start VPXD
service-control --stop vmware-vpxd
sleep 10
service-control --start vmware-vpxd

# Verify it came back
service-control --status vmware-vpxd

Full Service Stack Restart

# Nuclear option -- restarts all vCenter services
service-control --stop --all
sleep 15
service-control --start --all

# Verify all services
service-control --status --all

4.3 VPXD Log Analysis

Key Log Files

Check for Recent Errors

# Last 50 error entries in VPXD log
grep -i "error\|fatal\|panic\|exception" /var/log/vmware/vpxd/vpxd.log | tail -50

# Check for crash indicators
grep -c "core dump\|segfault\|SIGABRT" /var/log/vmware/vpxd/vpxd.log

# Check VPXD alert log
cat /var/log/vmware/vpxd/vpxd-alert.log | tail -20

Expected Output (Healthy): No recent fatal errors, no crash indicators, alert log empty or minimal entries.

5. vCenter Services Status

5.1 vmon-cli Service Listing

The vmon-cli utility manages vCenter services at the vMon (VMware Service Lifecycle Manager) level.

List All Services

vmon-cli --list

Expected Output (Healthy):

analyticss        STARTED
applmgmt          STARTED
certificateauthority STARTED
certificatemanagement STARTED
cis-license       STARTED
content-library   STARTED
eam               STARTED
envoy             STARTED
hvc               STARTED
imagebuilder      STARTED
infraprofile      STARTED
lookupsvc         STARTED
netdumper         STARTED
observability     STARTED
perfcharts        STARTED
pschealth         STARTED
rbd               STARTED
rhttpproxy        STARTED
sca               STARTED
sps               STARTED
statsmonitor      STARTED
sts               STARTED
topologysvc       STARTED
trustmanagement   STARTED
updatemgr         STARTED
vapi-endpoint     STARTED
vcha              STARTED
vlcm              STARTED
vmcam             STARTED
vmonapi           STARTED
vmware-vpostgres  STARTED
vpxd              STARTED
vpxd-svcs         STARTED
vsan-health       STARTED
vsm               STARTED
vsphere-ui        STARTED
vstats            STARTED
vtsdb             STARTED
wcp               STARTED

Check Specific Service Status

# Check a single service
vmon-cli --status vpxd

# Check multiple services
for svc in vpxd lookupsvc sts vmware-vpostgres vsphere-ui; do
  echo "$svc: $(vmon-cli --status $svc)"
done

5.2 service-control Commands

Check All Services Status

service-control --status --all

Expected Output (Healthy):

VMware vCenter Server:Status: RUNNING
VMware vAPI Endpoint:Status: RUNNING
VMware Content Library:Status: RUNNING
VMware Certificate Authority:Status: RUNNING
VMware Identity Management Service:Status: RUNNING
VMware Lookup Service:Status: RUNNING
VMware Security Token Service:Status: RUNNING
VMware vSphere Client:Status: RUNNING
VMware vSphere Update Manager:Status: RUNNING
VMware PostgreSQL:Status: RUNNING
VMware HTTP Reverse Proxy:Status: RUNNING
VMware Envoy Service:Status: RUNNING
...
(all services RUNNING)

Start / Stop / Restart Individual Services

# Stop a specific service
service-control --stop vmware-updatemgr

# Start a specific service
service-control --start vmware-updatemgr

# Restart a specific service (stop + start)
service-control --stop vmware-vsphere-ui && service-control --start vmware-vsphere-ui

5.3 Critical vs Non-Critical Services

6. vCenter Appliance Health (VAMI)

The VAMI (vCenter Server Appliance Management Interface) REST API provides health status for all key appliance subsystems. These endpoints return standardized color-coded health states: green, yellow, orange, red, gray.

6.1 System Health

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/system" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

Expected Output:

"green"

Detailed System Health (with messages)

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/system?messages=true" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq .

6.2 Memory Health

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/mem" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

Expected Output:

"green"

6.3 Storage Health

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/storage" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

Expected Output:

"green"

6.4 Database Storage Health

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/database-storage" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

Expected Output:

"green"

6.5 CPU Load Health

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/load" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

Expected Output:

"green"

6.6 Swap Health

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/swap" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

Expected Output:

"green"

6.7 Software Packages Health

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/softwarepackages" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

Expected Output:

"green"

Comprehensive VAMI Health Script

Run all health checks at once:

echo "=== vCenter Appliance Health Summary ==="
for endpoint in system mem storage database-storage load swap softwarepackages; do
  result=$(curl -sk -X GET \
    "https://${VC_FQDN}/api/appliance/health/${endpoint}" \
    -H "vmware-api-session-id: ${VC_TOKEN}" | tr -d '"')
  printf "%-25s : %s\n" "$endpoint" "$result"
done

Expected Output (All Healthy):

=== vCenter Appliance Health Summary ===
system                    : green
mem                       : green
storage                   : green
database-storage          : green
load                      : green
swap                      : green
softwarepackages          : green

7. Database Health

vCenter Server 8.x uses an embedded PostgreSQL database (vPostgres) for all configuration and inventory data. Database health is foundational to vCenter operations.

7.1 PostgreSQL Embedded DB Status

Service Status

# Check vPostgres service
systemctl status vmware-vpostgres

Expected Output (Healthy):

● vmware-vpostgres.service - VMware Postgres
   Loaded: loaded (/usr/lib/systemd/system/vmware-vpostgres.service; enabled)
   Active: active (running) since Mon 2026-03-23 10:14:55 UTC; 3 days ago
 Main PID: 4821 (postgres)
    Tasks: 48
   Memory: 512.3M
   CGroup: /system.slice/vmware-vpostgres.service
           ├─4821 /opt/vmware/vpostgres/current/bin/postgres -D /storage/db/vpostgres
           ├─4910 postgres: checkpointer
           ├─4911 postgres: background writer
           └─ ...

Test Database Connectivity

# Connect to VCDB and run a basic query
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT version();"

Expected Output:

                                                  version
------------------------------------------------------------------------------------------------------------
 PostgreSQL 14.x (VMware Postgres 14.x) on x86_64-unknown-linux-gnu, compiled by gcc ...
(1 row)

7.2 VCDB Size Monitoring

# Check total VCDB size
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
  "SELECT pg_size_pretty(pg_database_size('VCDB')) AS db_size;"

Expected Output:

 db_size
---------
 12 GB
(1 row)

Top Tables by Size

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT
  schemaname || '.' || tablename AS table_full_name,
  pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) AS total_size
FROM pg_tables
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
ORDER BY pg_total_relation_size(schemaname || '.' || tablename) DESC
LIMIT 15;"

Expected Output:

         table_full_name          | total_size
----------------------------------+------------
 vc.vpx_event_arg                 | 3200 MB
 vc.vpx_event                     | 2100 MB
 vc.vpx_task_event                | 1800 MB
 vc.vpx_stat_counter              | 980 MB
 vc.vpx_task                      | 850 MB
 ...
(15 rows)

7.3 Vacuum & Maintenance

Check Last Vacuum Time

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT
  schemaname || '.' || relname AS table_name,
  last_vacuum,
  last_autovacuum,
  last_analyze,
  n_dead_tup
FROM pg_stat_user_tables
WHERE n_dead_tup > 1000
ORDER BY n_dead_tup DESC
LIMIT 10;"

Expected Output:

       table_name        |      last_vacuum      |   last_autovacuum    |     last_analyze      | n_dead_tup
-------------------------+-----------------------+----------------------+-----------------------+------------
 vc.vpx_event_arg        | 2026-03-25 02:00:01   | 2026-03-25 14:30:22  | 2026-03-25 02:00:01   |       2341
 vc.vpx_event            | 2026-03-25 02:00:01   | 2026-03-25 14:28:11  | 2026-03-25 02:00:01   |       1822
 ...

Manual Vacuum (if needed)

# Standard vacuum (non-blocking)
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "VACUUM ANALYZE;"

# Full vacuum (blocking, reclaims space)
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "VACUUM FULL ANALYZE;"

8. vCenter HA (VCHA)

vCenter High Availability (VCHA) provides automated failover for vCenter Server using an active/passive/witness architecture.

8.1 VCHA Mode & Status

API Check

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/vcha/cluster/mode" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq .

Expected Output (VCHA Enabled):

{
  "mode": "ENABLED"
}

Get Full VCHA Cluster Status

curl -sk -X POST \
  "https://${VC_FQDN}/api/vcenter/vcha/cluster?action=get" \
  -H "vmware-api-session-id: ${VC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"partial": false}' | jq .

Expected Output (Healthy VCHA):

{
  "config_state": "CONFIGURED",
  "mode": "ENABLED",
  "health_state": "HEALTHY",
  "node1": {
    "state": "UP",
    "role": "ACTIVE",
    "runtime": {
      "ip": {
        "ipv4": { "address": "10.0.0.101" }
      }
    }
  },
  "node2": {
    "state": "UP",
    "role": "PASSIVE",
    "runtime": {
      "ip": {
        "ipv4": { "address": "10.0.0.102" }
      }
    }
  },
  "witness": {
    "state": "UP",
    "runtime": {
      "ip": {
        "ipv4": { "address": "10.0.0.103" }
      }
    }
  }
}

8.2 Active / Passive / Witness Status

CLI Check (from Active Node)

# Check VCHA state via CLI
/usr/lib/vmware-vcha/vcha-cli cluster-get-state

Expected Output:

VCHA Cluster State: HEALTHY
Active Node State: UP
Passive Node State: UP
Witness Node State: UP
Replication State: IN_SYNC

Database Replication Lag

# Check replication lag from active node
/opt/vmware/vpostgres/current/bin/psql -U postgres -c \
  "SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn,
   (sent_lsn - replay_lsn) AS replication_lag
   FROM pg_stat_replication;"

8.3 Failover Readiness

Manual Failover Test (Planned)

# Initiate planned failover via API
curl -sk -X POST \
  "https://${VC_FQDN}/api/vcenter/vcha/cluster?action=failover" \
  -H "vmware-api-session-id: ${VC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"planned": true}'

9. Certificate Health

Certificate expiry is one of the most common causes of vCenter outages. Regular monitoring of all certificate stores is essential.

9.1 Certificate Manager Tool

# Launch Certificate Manager (interactive)
/usr/lib/vmware-vmca/bin/certificate-manager

This interactive tool provides options:

1. Replace Machine SSL certificate with Custom Certificate
2. Replace VMCA Root certificate with Custom Signing Certificate
3. Replace Machine SSL certificate with VMCA Certificate
4. Regenerate a new VMCA Root Certificate
5. Replace Solution user certificates with Custom Certificate
6. Replace Solution user certificates with VMCA certificates
7. Revert last performed operation
8. Reset all Certificates

9.2 VECS Store Listing

List All VECS Stores

/usr/lib/vmware-vmafd/bin/vecs-cli store list

Expected Output:

MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vsphere-webclient
vpxd
vpxd-extension
hvc
data-encipherment
APPLMGMT_PASSWORD
SMS
wcp
backup_store

List Certificates in a Store

# List Machine SSL certificate
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT

Expected Output:

Alias : __MACHINE_CERT
Entry type : Private Key

Get Certificate Details from a Store

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert \
  --store MACHINE_SSL_CERT --alias __MACHINE_CERT | \
  openssl x509 -noout -subject -issuer -dates -serial

Expected Output:

subject=CN = vcenter01.lab.local
issuer=CN = CA, DC = vsphere, DC = local, C = US, ST = California, O = lab.local
notBefore=Jan 15 00:00:00 2026 GMT
notAfter=Jan 15 00:00:00 2028 GMT
serial=3A4B5C6D7E8F

9.3 STS Certificate

The Security Token Service (STS) signing certificate is critical for SSO authentication.

Check STS Certificate Expiry

# Extract and check the STS signing certificate
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list \
  --login administrator@vsphere.local \
  --password "${VC_PASS}" | head -20

Alternative: Check via Lookup Service

# Get STS certificate from LDAP
/usr/lib/vmware-vmdir/bin/ldapsearch -h localhost -p 389 \
  -b "cn=TenantCredential-1,cn=local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local" \
  -D "cn=administrator,cn=users,dc=vsphere,dc=local" \
  -w "${VC_PASS}" \
  userCertificate 2>/dev/null | grep -A1 "userCertificate"

Check STS Token Signing Certificate with Python Script

# VMware-provided STS cert check script
python /usr/lib/vmware-vmca/share/config/checksts.py

Expected Output (Healthy):

STS signing certificate:
  Subject: CN=ssoserver-sign
  Not Before: Jan 15 00:00:00 2026 GMT
  Not After:  Jan 15 00:00:00 2028 GMT
  Days remaining: 661
  Status: VALID

9.4 Machine SSL Certificate

# Check Machine SSL cert expiry remotely
echo | openssl s_client -connect ${VC_FQDN}:443 -servername ${VC_FQDN} 2>/dev/null | \
  openssl x509 -noout -subject -issuer -dates -checkend 2592000

Expected Output (Healthy):

subject=CN = vcenter01.lab.local
issuer=CN = CA, DC = vsphere, DC = local
notBefore=Jan 15 00:00:00 2026 GMT
notAfter=Jan 15 00:00:00 2028 GMT
Certificate will not expire

Check Expiry of All Solution User Certificates

for store in machine vsphere-webclient vpxd vpxd-extension; do
  echo "=== Store: ${store} ==="
  /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert \
    --store ${store} --alias ${store} 2>/dev/null | \
    openssl x509 -noout -subject -dates 2>/dev/null
done

9.5 Expiry Checks -- All Certificates

Comprehensive Certificate Expiry Report

# Check all VECS stores for certificate expiry
for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do
  for alias in $(/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store ${store} 2>/dev/null | grep "Alias" | awk '{print $3}'); do
    CERT=$(/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store ${store} --alias ${alias} 2>/dev/null)
    if [ -n "$CERT" ]; then
      EXPIRY=$(echo "$CERT" | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
      DAYS=$(echo "$CERT" | openssl x509 -noout -checkend 0 2>/dev/null && echo "VALID" || echo "EXPIRED")
      printf "Store: %-25s Alias: %-25s Expires: %-30s Status: %s\n" "$store" "$alias" "$EXPIRY" "$DAYS"
    fi
  done
done

API-based Certificate Check

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/certificate-management/vcenter/tls" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq '{
    subject: .subject_dn,
    issuer: .issuer_dn,
    valid_from: .valid_from,
    valid_to: .valid_to
  }'

Expected Output:

{
  "subject": "CN=vcenter01.lab.local",
  "issuer": "CN=CA, DC=vsphere, DC=local",
  "valid_from": "2026-01-15T00:00:00.000Z",
  "valid_to": "2028-01-15T00:00:00.000Z"
}

10. Storage Health

10.1 Disk Partitions & Filesystem

Check All Partitions

df -h

Expected Output (Healthy):

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        11G  4.2G  6.1G  41% /
tmpfs           6.0G   48M  5.9G   1% /dev/shm
/dev/sda1       128M   32M   97M  25% /boot
/dev/sda5        25G  5.8G   18G  25% /storage/log
/dev/sda6        10G  2.1G  7.4G  22% /storage/db
/dev/sda8        50G  9.2G   38G  20% /storage/seat
/dev/sda9        25G  3.3G   20G  15% /storage/netdump
/dev/sda10       10G  1.2G  8.2G  13% /storage/autodeploy
/dev/sda11       10G  836M  8.6G   9% /storage/imagebuilder
/dev/sda12       10G  2.5G  7.0G  27% /storage/updatemgr
/dev/sda13        5G   63M  4.7G   2% /storage/lifecycle

Check inode Usage

df -ih

10.2 Log Storage Utilization

# Check /storage/log usage
du -sh /storage/log/* 2>/dev/null | sort -rh | head -15

Expected Output:

2.1G    /storage/log/vmware/vpxd
812M    /storage/log/vmware/vsphere-ui
543M    /storage/log/vmware/sso
322M    /storage/log/vmware/eam
210M    /storage/log/vmware/rhttpproxy
...

Find Large Log Files

find /storage/log -type f -size +100M -exec ls -lh {} \; 2>/dev/null

10.3 DB Storage Utilization

# Check /storage/db usage
du -sh /storage/db/*

Expected Output:

8.2G    /storage/db/vpostgres

# Check PostgreSQL WAL files
du -sh /storage/db/vpostgres/pg_wal/

11. Performance & Resource Utilization

11.1 CPU Utilization

API Check

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/load" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

CLI Check

# Current CPU load
uptime

Expected Output:

 14:22:33 up 3 days,  4:07,  1 user,  load average: 1.23, 1.45, 1.32

# Top CPU consumers
top -bn1 | head -20

# CPU info
nproc
cat /proc/cpuinfo | grep "model name" | head -1

11.2 Memory Utilization

API Check

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/health/mem" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

CLI Check

free -m

Expected Output (Healthy):

              total        used        free      shared  buff/cache   available
Mem:          24576       14234        2180         312        8162        9876
Swap:          3071          32        3039

# Memory usage percentage
free -m | awk 'NR==2{printf "Memory Usage: %.1f%%\n", $3*100/$2}'

11.3 Swap & Load Average

# Swap usage
swapon --show

Expected Output (Healthy):

NAME      TYPE SIZE USED PRIO
/dev/sda2 partition 3G  32M   -2

# Detailed swap info
cat /proc/swaps
vmstat 1 5

12. SSO / Identity Source Health

12.1 SSO Domain Health

Check SSO Domain Status

# List SSO domains
/opt/vmware/bin/sso-config.sh -get_identity_sources

Expected Output:

Identity Source: vsphere.local
Type: System Domain
Default: true

Identity Source: lab.local
Type: ActiveDirectory
Default: false

Verify SSO Configuration

/opt/vmware/bin/sso-config.sh -get_default_identity_sources

12.2 Identity Source Connectivity

API Check

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/identity/providers" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq .

Test Authentication

# Test SSO login via API session creation
curl -sk -w "\nHTTP_CODE: %{http_code}\n" -X POST \
  "https://${VC_FQDN}/api/session" \
  -u "administrator@vsphere.local:${VC_PASS}"

Expected Output (Healthy):

"b1a2c3d4-e5f6-7890-abcd-ef1234567890"
HTTP_CODE: 201

12.3 LDAP / AD Binding Test

Test LDAP Connectivity

# Test LDAP bind to AD (if AD identity source configured)
ldapsearch -h dc01.lab.local -p 389 \
  -D "CN=svc_vcenter,OU=Service Accounts,DC=lab,DC=local" \
  -w 'ServiceAccountPassword' \
  -b "DC=lab,DC=local" \
  -s base "(objectClass=*)" 2>&1 | head -5

Expected Output (Healthy):

# extended LDIF
#
# LDAPv3
# base <DC=lab,DC=local> with scope baseObject
# filter: (objectClass=*)

Check VMware Directory Service (vmdir)

# Check vmdir service status
systemctl status vmware-stsd
/opt/vmware/bin/ldapsearch -h localhost -p 389 \
  -b "" -s base "(objectClass=*)" namingContexts 2>/dev/null

12.4 Token Validation

# Verify existing session token is valid
curl -sk -X GET \
  "https://${VC_FQDN}/api/session" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq .

Expected Output (Valid Token):

{
  "user": "VSPHERE.LOCAL\\Administrator",
  "created_time": "2026-03-26T10:15:22.000Z"
}

13. Plugins & Extensions

13.1 Registered Plugins

API Check

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/namespace-management/software/registries" \
  -H "vmware-api-session-id: ${VC_TOKEN}" 2>/dev/null | jq .

MOB (Managed Object Browser) Method

Navigate to: https://$VC_FQDN/mob/?moid=ExtensionManager&doPath=extensionList

PowerCLI Method

Connect-VIServer -Server $VC_FQDN -User $VC_USER -Password $VC_PASS
$em = Get-View ExtensionManager
$em.ExtensionList | Select-Object Key, Description, Company, Version | Format-Table -AutoSize

Expected Output:

Key                                      Description              Company              Version
---                                      -----------              -------              -------
com.vmware.vim.sms                       Storage Monitoring       VMware, Inc.         8.0.3
com.vmware.vcIntegrity                   vSphere Lifecycle Mgr    VMware, Inc.         8.0.3
com.vmware.vim.eam                       ESX Agent Manager        VMware, Inc.         8.0.3
com.vmware.rbd                           RBD                      VMware, Inc.         8.0.3
com.vmware.h4.vsphere.client             vSphere Client           VMware, Inc.         8.0.3
com.vmware.nsx.management.nsxt           NSX                      VMware, Inc.         4.2.1
...

13.2 Plugin Health Verification

# Check plugin health via REST
curl -sk -X GET \
  "https://${VC_FQDN}/ui/extensionmanager/extensionlist" \
  -H "vmware-api-session-id: ${VC_TOKEN}" 2>/dev/null | jq '.[].key'

Check for Plugin Load Errors

# Check vsphere-ui logs for plugin errors
grep -i "plugin\|extension" /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log | \
  grep -i "error\|fail\|exception" | tail -20

13.3 Stale Plugin Cleanup

Identify Stale Plugins

# Check for plugins whose server URL is unreachable
curl -sk -X GET \
  "https://${VC_FQDN}/mob/?moid=ExtensionManager" \
  -H "vmware-api-session-id: ${VC_TOKEN}" 2>/dev/null

Remove Stale Plugin via MOB

1. Navigate to: https://$VC_FQDN/mob/?moid=ExtensionManager
2. Click UnregisterExtension
3. Enter the extension key (e.g., com.vendor.stale.plugin)
4. Click Invoke Method

Remove via PowerCLI

$em = Get-View ExtensionManager
$em.UnregisterExtension("com.vendor.stale.plugin")

14. Lookup Service & PSC

The Lookup Service is the service registry for all vSphere components. Since vCenter 7.0+, the Platform Services Controller (PSC) is embedded.

14.1 Lookup Service Registration

Check Lookup Service Status

service-control --status vmware-lookupsvc

Expected Output:

VMware Lookup Service:Status: RUNNING

List All Service Registrations

# Use lstool to list registrations
python /usr/lib/vmidentity/tools/scripts/lstool.py list \
  --url "https://localhost/lookupservice/sdk" \
  --no-check-cert 2>/dev/null | head -60

Expected Output (Healthy):

=== Service Registration ===
Service ID: vcenterserver
Owner ID: vcenter01.lab.local@vsphere.local
Service Type: vcenterserver
Endpoints:
  URL: https://vcenter01.lab.local/sdk
  Protocol: vmomi

Service ID: cs.identity
Owner ID: vcenter01.lab.local@vsphere.local
Service Type: cs.identity
Endpoints:
  URL: https://vcenter01.lab.local/sts/STSService/vsphere.local
  Protocol: wsTrust
...

14.2 STS Health

# Check STS service status
service-control --status vmware-stsd

Expected Output:

VMware Security Token Service:Status: RUNNING

Test STS Token Issuance

# Verify STS can issue tokens by creating an API session
curl -sk -X POST \
  "https://${VC_FQDN}/api/session" \
  -u "${VC_USER}:${VC_PASS}" \
  -w "\nHTTP Status: %{http_code}\n"

14.3 Service Registration Entries

Verify Key Registrations Exist

python /usr/lib/vmidentity/tools/scripts/lstool.py list \
  --url "https://localhost/lookupservice/sdk" \
  --no-check-cert 2>/dev/null | grep "Service Type" | sort -u

Expected Service Types:

Service Type: cs.authorization
Service Type: cs.identity
Service Type: cs.license
Service Type: cs.lookup
Service Type: cs.privilege
Service Type: sso:admin
Service Type: sso:groupcheck
Service Type: sso:sts
Service Type: vcenterserver
Service Type: cs.inventory
Service Type: cs.envoy

15. Inventory Verification

15.1 Datacenter / Cluster / Host / VM Counts

Get Datacenter Count

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/datacenter" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length'

List All Datacenters

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/datacenter" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq '.[] | {name, datacenter}'

Expected Output:

{
  "name": "DC-Site-A",
  "datacenter": "datacenter-1"
}
{
  "name": "DC-Site-B",
  "datacenter": "datacenter-2"
}

Get Cluster Count and List

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/cluster" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq '.[] | {name, cluster, ha_enabled, drs_enabled}'

Expected Output:

{
  "name": "Management-Cluster",
  "cluster": "domain-c8",
  "ha_enabled": true,
  "drs_enabled": true
}
{
  "name": "Workload-Cluster-01",
  "cluster": "domain-c44",
  "ha_enabled": true,
  "drs_enabled": true
}

Get Host Count and Status

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/host" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq '.[] | {name, host, connection_state, power_state}'

Expected Output:

{
  "name": "esxi01.lab.local",
  "host": "host-10",
  "connection_state": "CONNECTED",
  "power_state": "POWERED_ON"
}
{
  "name": "esxi02.lab.local",
  "host": "host-11",
  "connection_state": "CONNECTED",
  "power_state": "POWERED_ON"
}

Get VM Count

# Total VM count
curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/vm" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length'

# Powered-on VMs
curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/vm?power_states=POWERED_ON" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length'

Full Inventory Summary Script

echo "=== vCenter Inventory Summary ==="
DC=$(curl -sk "https://${VC_FQDN}/api/vcenter/datacenter" -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length')
CL=$(curl -sk "https://${VC_FQDN}/api/vcenter/cluster" -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length')
HO=$(curl -sk "https://${VC_FQDN}/api/vcenter/host" -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length')
VM=$(curl -sk "https://${VC_FQDN}/api/vcenter/vm" -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length')
VMon=$(curl -sk "https://${VC_FQDN}/api/vcenter/vm?power_states=POWERED_ON" -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length')
DS=$(curl -sk "https://${VC_FQDN}/api/vcenter/datastore" -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length')
NET=$(curl -sk "https://${VC_FQDN}/api/vcenter/network" -H "vmware-api-session-id: ${VC_TOKEN}" | jq 'length')
echo "Datacenters:  ${DC}"
echo "Clusters:     ${CL}"
echo "Hosts:        ${HO}"
echo "Total VMs:    ${VM}"
echo "Powered-On:   ${VMon}"
echo "Datastores:   ${DS}"
echo "Networks:     ${NET}"

15.2 Inventory Consistency

Check for Disconnected Hosts

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/host?connection_states=DISCONNECTED,NOT_RESPONDING" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq '.[] | {name, connection_state}'

Expected Output (Healthy): Empty array []

Check for Orphaned VMs

curl -sk -X GET \
  "https://${VC_FQDN}/api/vcenter/vm?power_states=POWERED_OFF" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq '.[] | {name, vm, power_state}'

16. Syslog & Log Configuration

16.1 Syslog Forwarding

Check Syslog Configuration via API

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/logging/forwarding" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq .

Expected Output (Configured):

[
  {
    "hostname": "syslog.lab.local",
    "port": 514,
    "protocol": "UDP"
  },
  {
    "hostname": "loginsight.lab.local",
    "port": 9000,
    "protocol": "TCP"
  }
]

Test Syslog Forwarding

curl -sk -X POST \
  "https://${VC_FQDN}/api/appliance/logging/forwarding?action=test" \
  -H "vmware-api-session-id: ${VC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"send_test_message": true}' | jq .

Expected Output:

[
  {
    "hostname": "syslog.lab.local",
    "state": "UP",
    "message": ""
  }
]

Configure Syslog Forwarding

curl -sk -X PUT \
  "https://${VC_FQDN}/api/appliance/logging/forwarding" \
  -H "vmware-api-session-id: ${VC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '[{"hostname": "syslog.lab.local", "port": 514, "protocol": "UDP"}]'

16.2 Log Rotation

Check Logrotate Configuration

# Check logrotate status
cat /etc/logrotate.conf | head -20

# Check vCenter-specific rotation configs
ls -la /etc/logrotate.d/

Trigger Manual Log Rotation

logrotate -f /etc/logrotate.conf

16.3 Log Bundle Generation

Generate Support Bundle via API

curl -sk -X POST \
  "https://${VC_FQDN}/api/appliance/support-bundle" \
  -H "vmware-api-session-id: ${VC_TOKEN}" \
  -H "Content-Type: application/json" | jq .

Generate via CLI

# Generate support bundle
/usr/lib/vmware-vpxd/support/vcdb_report.sh

# Generate full log bundle
vc-support-bundle

17. NTP Configuration

Time synchronization is critical for vCenter operations, certificate validation, SSO token integrity, and log correlation.

17.1 Time Sync via API

Get NTP Configuration

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/ntp" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq .

Expected Output:

[
  "ntp1.lab.local",
  "ntp2.lab.local"
]

Get Time Sync Mode

curl -sk -X GET \
  "https://${VC_FQDN}/api/appliance/timesync" \
  -H "vmware-api-session-id: ${VC_TOKEN}"

Expected Output:

"NTP"

Set NTP Servers via API

curl -sk -X PUT \
  "https://${VC_FQDN}/api/appliance/ntp" \
  -H "vmware-api-session-id: ${VC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '["ntp1.lab.local", "ntp2.lab.local"]'

17.2 Time Sync via CLI

Check NTP Service

# Check NTP daemon status
systemctl status systemd-timesyncd

# Check NTP synchronization
timedatectl status

Expected Output (Healthy):

               Local time: Thu 2026-03-26 14:22:33 UTC
           Universal time: Thu 2026-03-26 14:22:33 UTC
                 RTC time: Thu 2026-03-26 14:22:33
                Time zone: UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

# Check NTP peers
ntpq -p 2>/dev/null || chronyc sources 2>/dev/null

Expected Output:

MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp1.lab.local                2   6   377    34   +0.234ms[+0.312ms] +/- 12ms
^+ ntp2.lab.local                2   6   377    35   +1.023ms[+1.101ms] +/- 15ms

17.3 Drift Check

# Check clock offset
chronyc tracking 2>/dev/null | grep "System time\|Last offset"

Expected Output:

System time     : 0.000000234 seconds fast of NTP time
Last offset     : +0.000000312 seconds

18. Port Reference Table

vCenter Server Inbound Ports

vCenter Server Outbound Ports

Port Verification Commands

# Check listening ports on vCenter appliance
ss -tlnp | sort -t: -k2 -n

# Check specific port connectivity
curl -sk -o /dev/null -w "%{http_code}" https://${VC_FQDN}:443
curl -sk -o /dev/null -w "%{http_code}" https://${VC_FQDN}:5480

# Check outbound connectivity
nc -zv ntp1.lab.local 123 2>&1
nc -zv dc01.lab.local 389 2>&1
nc -zv esxi01.lab.local 443 2>&1

19. Common Issues & Remediation

19.1 VPXD Crashes or Won't Start

Symptoms: vSphere Client inaccessible, service-control --status vmware-vpxd shows STOPPED.

Diagnostic Steps:

# Check VPXD logs for crash reason
tail -200 /var/log/vmware/vpxd/vpxd.log | grep -i "error\|fatal\|abort"

# Check for core dumps
ls -la /var/core/

# Check if database is reachable
systemctl status vmware-vpostgres

# Check disk space (VPXD won't start if disk full)
df -h /storage/log /storage/db /

# Check for port conflicts
ss -tlnp | grep ":443\b"

19.2 Database Corruption

Symptoms: VPXD errors referencing VCDB, inventory inconsistencies, SQL errors in logs.

# Check PostgreSQL logs
tail -100 /var/log/vmware/vpostgres/postgresql*.log

# Run database integrity check
/opt/vmware/vpostgres/current/bin/pg_isready -h localhost -p 5432

# Check for corruption indicators
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
  "SELECT datname, datallowconn FROM pg_database;"

19.3 Certificate Expiry

Symptoms: SSO login failures, "certificate expired" errors in browser, service communication failures.

# Quick check all certs
for store in MACHINE_SSL_CERT machine vsphere-webclient vpxd vpxd-extension; do
  echo "=== ${store} ==="
  /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store ${store} \
    --alias $(/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store ${store} 2>/dev/null | grep Alias | awk '{print $3}' | head -1) 2>/dev/null | \
    openssl x509 -noout -dates 2>/dev/null
done

# Check STS cert specifically
python /usr/lib/vmware-vmca/share/config/checksts.py 2>/dev/null

19.4 SSO Lockout

Symptoms: Cannot log in as administrator@vsphere.local, "invalid credentials" or "account locked."

# Check lockout policy
/opt/vmware/bin/sso-config.sh -get_lockout_policy

# Check failed login attempts
grep -i "login\|auth\|lock" /var/log/vmware/sso/ssoAdminServer.log | tail -30

19.5 Performance Degradation

Symptoms: vSphere Client slow, API timeouts, tasks taking excessively long.

# Identify bottleneck
echo "=== CPU ===" && uptime
echo "=== Memory ===" && free -m
echo "=== Swap ===" && swapon --show
echo "=== Disk I/O ===" && iostat -x 1 3 2>/dev/null || echo "iostat not available"
echo "=== Top Processes ===" && top -bn1 | head -15
echo "=== DB Connections ===" && /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
  "SELECT count(*) FROM pg_stat_activity;"

20. CLI Quick Reference Card

Service Management

System & Appliance

Database

Certificates

SSO & Identity

Lookup Service

Logs

VCHA

Networking

21. API Quick Reference

Authentication

Appliance Health

Appliance Configuration

Logging

Inventory

vCenter HA

Certificates

Identity & SSO

Common curl Pattern

# GET request pattern
curl -sk -X GET \
  "https://${VC_FQDN}/api/<endpoint>" \
  -H "vmware-api-session-id: ${VC_TOKEN}" | jq .

# POST request pattern
curl -sk -X POST \
  "https://${VC_FQDN}/api/<endpoint>" \
  -H "vmware-api-session-id: ${VC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{ "key": "value" }' | jq .

# PUT request pattern
curl -sk -X PUT \
  "https://${VC_FQDN}/api/<endpoint>" \
  -H "vmware-api-session-id: ${VC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{ "key": "value" }'

Session Management Best Practice

# Create session at start
export VC_TOKEN=$(curl -sk -X POST \
  "https://${VC_FQDN}/api/session" \
  -u "${VC_USER}:${VC_PASS}" | tr -d '"')

# ... run all health checks ...

# Destroy session at end
curl -sk -X DELETE \
  "https://${VC_FQDN}/api/session" \
  -H "vmware-api-session-id: ${VC_TOKEN}"
unset VC_TOKEN VC_PASS

(c) 2026 Virtual Control LLC. All rights reserved.