Fleet / SDDC Manager Health Check Handbook

1. Overview & Purpose

This handbook provides a complete health check procedure for the SDDC Manager (Fleet Manager) in a VCF 9.0 environment. SDDC Manager is the central orchestration and lifecycle management component of VCF. Its health directly affects your ability to:

• Deploy and manage workload domains
• Perform lifecycle updates (patches, upgrades)
• Commission/decommission ESXi hosts
• Manage certificates across all VCF components
• Monitor component inventory and compliance

When to Run This Health Check

2. Prerequisites

Required Access

Token Acquisition

export SDDC="sddc-manager.lab.local"
export SDDC_USER="admin@local"
export SDDC_PASS="YourPassword123!"

# Acquire access token
TOKEN=$(curl -sk -X POST "https://$SDDC/v1/tokens" \
  -H "Content-Type: application/json" \
  -d "{\"username\":\"$SDDC_USER\",\"password\":\"$SDDC_PASS\"}" | jq -r '.accessToken')

echo "Token: ${TOKEN:0:20}..."

# Convenience function
sddc_api() {
  curl -sk -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    "https://$SDDC$1" 2>/dev/null
}

3. Quick Reference — All Checks Summary

4. Service Status

4.1 All Services via systemctl

What: Verify all SDDC Manager services are running.

ssh vcf@$SDDC
# Then switch to root or use sudo:
sudo systemctl list-units --type=service --state=running | grep -E "vcf|sddc|operationsd|commonsvcs|domainmanager|lcm"

Expected Output:

  commonsvcs.service               loaded active running   VCF Common Services
  domainmanager.service            loaded active running   VCF Domain Manager
  lcm.service                      loaded active running   VCF Lifecycle Manager
  operationsd.service              loaded active running   VCF Operations
  sddc-manager-ui-app.service      loaded active running   SDDC Manager UI
  sddc-support.service             loaded active running   VCF Support Service

4.2 VCF Service Status Script

# Comprehensive service check
for SVC in commonsvcs domainmanager lcm operationsd sddc-manager-ui-app sddc-support; do
  STATUS=$(systemctl is-active $SVC 2>/dev/null)
  if [ "$STATUS" = "active" ]; then
    echo "[PASS] $SVC: $STATUS"
  else
    echo "[FAIL] $SVC: $STATUS"
  fi
done

4.3 Critical vs Non-Critical Services

5. Database Health

5.1 PostgreSQL Status

ssh root@$SDDC
systemctl status postgresql

Expected Output:

● postgresql.service - PostgreSQL database server
     Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled)
     Active: active (running) since Mon 2026-03-20 08:00:00 UTC; 6 days ago
   Main PID: 1234 (postgres)
     Memory: 256.0M

Connection Test

sudo -u postgres psql -c "SELECT version();"

Expected Output:

                                  version
---------------------------------------------------------------------------
 PostgreSQL 14.x on x86_64-pc-linux-gnu, compiled by gcc ...
(1 row)

5.2 Database Size & Connections

# Database size
sudo -u postgres psql -c "
SELECT datname, pg_size_pretty(pg_database_size(datname)) as size
FROM pg_database WHERE datname NOT IN ('template0','template1','postgres')
ORDER BY pg_database_size(datname) DESC;"

Expected Output:

    datname     |  size
-----------------+---------
 sddc_manager_db | 2.1 GB
 lcm_db          | 512 MB
 operations_db   | 256 MB

# Active connections
sudo -u postgres psql -c "
SELECT datname, count(*) as connections
FROM pg_stat_activity
GROUP BY datname ORDER BY connections DESC;"

5.3 Vacuum Status

sudo -u postgres psql -d sddc_manager_db -c "
SELECT schemaname, relname, n_dead_tup, last_autovacuum
FROM pg_stat_user_tables
WHERE n_dead_tup > 10000
ORDER BY n_dead_tup DESC LIMIT 10;"

5.4 Database Backup

# Check if database backups exist
ls -la /opt/vmware/vcf/sddc-manager/backup/ 2>/dev/null
ls -la /nfs-mount/vcf-backups/ 2>/dev/null

6. Component Inventory

6.1 System Components

What: Verify all VCF-managed components are in a healthy state.

# List all VCF components
sddc_api "/v1/system" | jq .

Get all vCenters

sddc_api "/v1/vcenter" | jq '.elements[] | {
  id: .id,
  fqdn: .fqdn,
  version: .version,
  status: .status
}'

Get all NSX Managers

sddc_api "/v1/nsxt-clusters" | jq '.elements[] | {
  id: .id,
  vipFqdn: .vipFqdn,
  version: .version
}'

6.2 Workload Domains

sddc_api "/v1/domains" | jq '.elements[] | {
  id: .id,
  name: .name,
  type: .type,
  status: .status
}'

Expected Output:

{
  "id": "abc123...",
  "name": "MGMT",
  "type": "MANAGEMENT",
  "status": "ACTIVE"
}
{
  "id": "def456...",
  "name": "WLD-01",
  "type": "VI",
  "status": "ACTIVE"
}

6.3 Host Inventory

sddc_api "/v1/hosts" | jq '.elements[] | {
  id: .id,
  fqdn: .fqdn,
  status: .status,
  domain: .domain.name
}'

7. Lifecycle Management (LCM)

7.1 Current Version

# SDDC Manager version
sddc_api "/v1/system" | jq '{
  version: .version,
  build: .build,
  fips_enabled: .fipsEnabled
}'

7.2 Available Updates

# Check available updates
sddc_api "/v1/system/prechecks" | jq .

Bundle Availability

sddc_api "/v1/bundles" | jq '.elements[] | {
  id: .id,
  bundleType: .bundleType,
  version: .version,
  status: .downloadStatus,
  components: [.components[].type]
}'

7.3 Upgrade Prechecks

# Trigger precheck
curl -sk -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  "https://$SDDC/v1/system/prechecks" \
  -d '{"bundleId":"<bundle-id>"}'

# Check precheck results
sddc_api "/v1/system/prechecks/<precheck-id>" | jq '.results[] | {
  check: .description,
  status: .status,
  severity: .severity,
  resolution: .resolution
}'

8. Bundle Management

# List all bundles
sddc_api "/v1/bundles" | jq '.elements[] | {
  id: .id,
  type: .bundleType,
  version: .version,
  downloadStatus: .downloadStatus,
  applicableVersions: .applicableVersions
}'

Download Statuses:

Check Depot Connectivity

# Test connectivity to VMware depot (online mode)
ssh vcf@$SDDC
curl -sk https://depot.vmware.com/PROD2/vcf/manifest.json | head -5

9. DNS & NTP Verification

DNS Configuration

# Get DNS config via API
sddc_api "/v1/system/dns-configuration" | jq .

Expected Output:

{
  "dnsServers": [
    {"ipAddress": "192.168.1.1", "isPrimary": true},
    {"ipAddress": "192.168.1.2", "isPrimary": false}
  ]
}

DNS Resolution Test

ssh vcf@$SDDC
# Test forward resolution for all VCF components
for HOST in vcenter.lab.local nsx-vip.lab.local sddc-manager.lab.local; do
  echo "$HOST: $(nslookup $HOST | grep Address | tail -1)"
done

# Test reverse resolution
for IP in 192.168.1.70 192.168.1.71 192.168.1.60; do
  echo "$IP: $(nslookup $IP | grep name | head -1)"
done

NTP Configuration

# Get NTP config
sddc_api "/v1/system/ntp-configuration" | jq .

# Check time sync on appliance
ssh vcf@$SDDC
timedatectl status
chronyc tracking

10. Certificate Health

# List all certificates managed by VCF
sddc_api "/v1/certificate-authorities" | jq .

# Get certificates for a specific resource
sddc_api "/v1/domains/<domain-id>/resource-certificates" | jq '.elements[] | {
  resource: .resourceFqdn,
  type: .certificateType,
  expiresAt: .expirationDate,
  issuedBy: .issuedBy
}'

Check Certificate Expiry via OpenSSL

# Check SDDC Manager certificate
echo | openssl s_client -connect $SDDC:443 2>/dev/null | \
  openssl x509 -noout -dates -subject

CSR Status

sddc_api "/v1/certificate-authorities/csr" | jq .

11. Task & Workflow History

# List recent tasks (last 20)
sddc_api "/v1/tasks?limit=20" | jq '.elements[] | {
  id: .id,
  name: .name,
  status: .status,
  type: .type,
  creationTimestamp: .creationTimestamp,
  completionTimestamp: .completionTimestamp
}'

Task Statuses:

Check for Failed Tasks

sddc_api "/v1/tasks?status=FAILED&limit=10" | jq '.elements[] | {
  id: .id,
  name: .name,
  creationTimestamp: .creationTimestamp,
  errorMessage: .errors
}'

Retry a Failed Task

curl -sk -X PATCH -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  "https://$SDDC/v1/tasks/<task-id>" \
  -d '{"status":"IN_PROGRESS"}'

12. Host Commissioning Status

# All hosts with their status
sddc_api "/v1/hosts" | jq '.elements[] | {
  fqdn: .fqdn,
  status: .status,
  domain: .domain.name,
  clusterId: .cluster.id
}'

# Count by status
sddc_api "/v1/hosts" | jq '[.elements[] | .status] | group_by(.) | map({status: .[0], count: length})'

13. Workload Domain Health

# List all domains with health
sddc_api "/v1/domains" | jq '.elements[] | {
  name: .name,
  type: .type,
  status: .status,
  clusters: [.clusters[].name]
}'

Cluster Health per Domain

# List clusters
sddc_api "/v1/clusters" | jq '.elements[] | {
  name: .name,
  status: .status,
  domain: .domainName,
  hostCount: (.hosts | length),
  primaryDatastore: .primaryDatastoreType
}'

14. Backup & Restore

# Get backup configuration
sddc_api "/v1/system/backup-configuration" | jq .

Expected Output:

{
  "backupEnabled": true,
  "backupSchedule": {
    "frequency": "DAILY",
    "daysOfWeek": null,
    "hourOfDay": 2,
    "minuteOfHour": 0
  },
  "server": {
    "protocol": "SFTP",
    "host": "backup-server.lab.local",
    "port": 22,
    "directory": "/backups/sddc/"
  },
  "encryption": {
    "passphrase": "***"
  }
}

Check Backup History

sddc_api "/v1/system/backup-configuration/backups" | jq '.elements[0:3]'

15. API Health Verification

Token Acquisition Test

# Time the token acquisition
time curl -sk -X POST "https://$SDDC/v1/tokens" \
  -H "Content-Type: application/json" \
  -d "{\"username\":\"$SDDC_USER\",\"password\":\"$SDDC_PASS\"}" | jq -r '.accessToken' > /dev/null

Endpoint Health Check Script

ENDPOINTS="/v1/system /v1/domains /v1/hosts /v1/clusters /v1/bundles"
for EP in $ENDPOINTS; do
  START=$(date +%s%N)
  HTTP_CODE=$(curl -sk -o /dev/null -w "%{http_code}" \
    -H "Authorization: Bearer $TOKEN" \
    "https://$SDDC$EP")
  END=$(date +%s%N)
  DURATION=$(( (END - START) / 1000000 ))
  echo "$EP: HTTP $HTTP_CODE (${DURATION}ms)"
done

16. Resource Utilization

ssh root@$SDDC

# CPU
uptime
top -b -n 1 | head -5

# Memory
free -m

# Disk
df -h

Critical Partitions

17. Port Reference Table

Inbound Ports

Outbound Ports

18. Common Issues & Remediation

18.1 Service Failures

18.2 Database Issues

18.3 LCM Failures

18.4 Task Stuck in IN_PROGRESS

# Find stuck tasks (running > 2 hours)
sddc_api "/v1/tasks?status=IN_PROGRESS" | jq '.elements[] | {
  id: .id,
  name: .name,
  creationTimestamp: .creationTimestamp
}'

18.5 Certificate Problems

19. CLI Quick Reference Card

Service Management

Database Commands

System Commands

Log Files

20. API Quick Reference

Authentication

# Acquire token
curl -sk -X POST "https://$SDDC/v1/tokens" \
  -H "Content-Type: application/json" \
  -d '{"username":"admin@local","password":"password"}'

# Use token
curl -sk -H "Authorization: Bearer $TOKEN" "https://$SDDC/v1/..."

Key Endpoints

Common Query Parameters