VCF Operations Health Check Handbook

1. Overview & Purpose

This handbook provides a complete health check procedure for VCF Operations (formerly VMware Aria Operations / vRealize Operations) deployed within a VCF 9.0 environment. VCF Operations provides:

• Performance monitoring — Real-time and historical analytics for all VCF components
• Capacity management — Predictive capacity analysis and right-sizing recommendations
• Alerting — Proactive notification of health, risk, and efficiency issues
• Compliance — Regulatory and best-practice compliance dashboards
• Troubleshooting — Root cause analysis and workload optimization

When to Run

2. Prerequisites

Required Access

Token Acquisition (Suite API)

export OPS="vcf-ops.lab.local"
export OPS_USER="admin"
export OPS_PASS="YourPassword123!"

# Acquire auth token
OPS_TOKEN=$(curl -sk -X POST \
  "https://$OPS/suite-api/api/auth/token/acquire" \
  -H "Content-Type: application/json" \
  -d "{
    \"username\":\"$OPS_USER\",
    \"password\":\"$OPS_PASS\",
    \"authSource\":\"local\"
  }" | jq -r '.token')

echo "Token: ${OPS_TOKEN:0:20}..."

# Convenience function
ops_api() {
  curl -sk -H "Authorization: vRealizeOpsToken $OPS_TOKEN" \
    -H "Content-Type: application/json" \
    "https://$OPS/suite-api$1" 2>/dev/null
}

3. Quick Reference — All Checks Summary

4. Analytics Cluster Status

4.1 Cluster State via CASA

What: Verify the VCF Operations analytics cluster is fully initialized and running.

Why: A cluster not in RUNNING state means data collection, alerting, and dashboards may be stale or non-functional.

SSH Method (on master node)

ssh root@$OPS

# Check cluster status via CASA admin
$VMWARE_PYTHON_PATH/bin/python \
  /usr/lib/vmware-vcops/tools/opscli/admin-cli.py \
  getClusterStatus

Expected Output (Healthy):

Cluster Status: RUNNING
Cluster Uptime: 15 days 8 hours
Master Node: vcf-ops-01.lab.local (ONLINE)
Data Node: vcf-ops-02.lab.local (ONLINE)
Data Node: vcf-ops-03.lab.local (ONLINE)
Remote Collector: rc-01.lab.local (ONLINE)

Alternative — CASA API

curl -sk "https://$OPS/casa/cluster/status" \
  -u "admin:$OPS_PASS" | jq .

Expected Output:

{
  "cluster_status": "RUNNING",
  "slice_status": "ONLINE",
  "node_statuses": [
    {"node_name": "vcf-ops-01", "status": "ONLINE", "role": "MASTER"},
    {"node_name": "vcf-ops-02", "status": "ONLINE", "role": "DATA"},
    {"node_name": "vcf-ops-03", "status": "ONLINE", "role": "DATA"}
  ]
}

Pass / Warn / Fail

4.2 Slice Status

What: Verify all analytics slices are online.

# Via CASA API
curl -sk "https://$OPS/casa/slice/status" \
  -u "admin:$OPS_PASS" | jq .

Expected Output:

{
  "slices": [
    {"slice_id": 0, "status": "ONLINE", "node": "vcf-ops-01"},
    {"slice_id": 1, "status": "ONLINE", "node": "vcf-ops-02"},
    {"slice_id": 2, "status": "ONLINE", "node": "vcf-ops-03"}
  ]
}

4.3 Node Roles

5. Node Health

5.1 Individual Node Status

# List all nodes via Suite API
ops_api "/api/deployment/node" | jq '.nodeList[] | {
  name: .name,
  ip: .ip,
  role: .role,
  status: .status,
  version: .version
}'

5.2 Resource Utilization per Node

ssh root@$OPS

# CPU
top -b -n 1 | head -5

# Memory
free -m

# Disk (critical partitions)
df -h /storage /storage/db /storage/log /storage/core

Critical Partitions

5.3 Heartbeat Verification

# Check last heartbeat per node
ops_api "/api/deployment/node" | jq '.nodeList[] | {
  name: .name,
  lastHeartbeat: .lastHeartbeat,
  heartbeatStatus: .heartbeatStatus
}'

6. Adapter Health

6.1 Adapter Instances

What: Verify all configured adapter instances are collecting data.

# List all adapters
ops_api "/api/adapters" | jq '.adapterInstancesInfoDto[] | {
  id: .id,
  adapterKind: .resourceKey.adapterKindKey,
  name: .resourceKey.name,
  collectorId: .collectorId,
  collectionState: .collectionState,
  collectionStatus: .collectionStatus
}'

Expected Output:

{
  "id": "abc123",
  "adapterKind": "VMWARE",
  "name": "vCenter - vcenter.lab.local",
  "collectorId": "1",
  "collectionState": "COLLECTING",
  "collectionStatus": "DATA_RECEIVING"
}

Key Adapters to Verify

6.2 Collection Status

# Check last collection time for a specific adapter
ADAPTER_ID="<adapter-id>"
ops_api "/api/adapters/$ADAPTER_ID" | jq '{
  name: .resourceKey.name,
  collectionState: .collectionState,
  lastCollected: .lastCollected,
  numberOfMetricsCollected: .numberOfMetricsCollected,
  numberOfResourcesCollected: .numberOfResourcesCollected
}'

6.3 Credential Validation

# List credentials
ops_api "/api/credentials" | jq '.credentialInstances[] | {
  id: .id,
  name: .name,
  adapterKind: .adapterKindKey
}'

# Test credential (trigger validation)
curl -sk -X POST \
  -H "Authorization: vRealizeOpsToken $OPS_TOKEN" \
  -H "Content-Type: application/json" \
  "https://$OPS/suite-api/api/adapters/$ADAPTER_ID/monitoringstate/start"

7. Collection Status

What: Verify data collection is current and no gaps exist.

# Get collection stats
ops_api "/api/resources?adapterKind=VMWARE&resourceKind=VirtualMachine&pageSize=5" | jq '{
  totalCount: .totalCount,
  resources: [.resourceList[].resourceKey.name]
}'

Check for Collection Gaps

# Recent collection cycles on the node
ssh root@$OPS
grep "Collection completed" /storage/log/vcops/analytics/analytics.log | tail -10

8. Certificate Health

# Check web certificate
echo | openssl s_client -connect $OPS:443 2>/dev/null | \
  openssl x509 -noout -dates -subject

# Check all certificates on the node
ssh root@$OPS
find /storage/vcops/user/conf/ssl -name "*.pem" -exec \
  sh -c 'echo "=== $1 ===" && openssl x509 -in "$1" -noout -enddate' _ {} \;

9. Capacity & Licensing

# Check license status
ops_api "/api/deployment/licenses" | jq '.licenseDetails[] | {
  licenseKey: .licenseKey[0:8],
  edition: .edition,
  capacity: .capacity,
  usage: .usage,
  expirationDate: .expirationDate
}'

Expected Output:

{
  "licenseKey": "XXXXX-XX",
  "edition": "Enterprise",
  "capacity": 500,
  "usage": 320,
  "expirationDate": "2027-03-01"
}

10. Disk & Database Health

Disk Usage

ssh root@$OPS
df -h /storage /storage/db /storage/log

Cassandra / xDB Health

# Check Cassandra status
ssh root@$OPS
/opt/vmware/vcops/cassandra/apache-cassandra/bin/nodetool status

Expected Output:

Datacenter: vrops
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns    Host ID
UN  192.168.1.77    15.2 GiB   256     100.0%  abc123...
UN  192.168.1.78    14.8 GiB   256     100.0%  def456...
UN  192.168.1.79    15.0 GiB   256     100.0%  ghi789...

Data Retention

# Check retention settings
ops_api "/api/deployment/retention" | jq .

11. Alert Engine Health

# Count active alerts by criticality
ops_api "/api/alerts?status=ACTIVE&criticality=CRITICAL" | jq '.totalCount'
ops_api "/api/alerts?status=ACTIVE&criticality=IMMEDIATE" | jq '.totalCount'
ops_api "/api/alerts?status=ACTIVE&criticality=WARNING" | jq '.totalCount'

Check Alert Plugins (Notifications)

ops_api "/api/alertplugins" | jq '.notificationPluginInstances[] | {
  id: .id,
  name: .name,
  pluginType: .pluginTypeId,
  enabled: .enabled
}'

Test SMTP Notification

# Verify SMTP relay
ssh root@$OPS
echo "Test" | mail -s "VCF Ops Health Check Test" admin@lab.local

12. Remote Collectors

# List remote collectors
ops_api "/api/collectors" | jq '.collector[] | {
  id: .id,
  name: .name,
  ip: .ip,
  status: .state,
  version: .version,
  usingVRealize: .usingVRealize
}'

Expected Output:

{
  "id": "1",
  "name": "vcf-ops-rc-01",
  "ip": "192.168.1.80",
  "status": "ONLINE",
  "version": "8.18.0.12345678"
}

13. Management Packs

# List installed management packs (solutions)
ops_api "/api/solutions" | jq '.solution[] | {
  id: .id,
  name: .name,
  version: .version,
  adapterKind: .adapterKindKeys
}'

14. Integration Health

vCenter Adapter

# Check vCenter adapter specifically
ops_api "/api/adapters?adapterKindKey=VMWARE" | jq '.adapterInstancesInfoDto[] | {
  name: .resourceKey.name,
  collectionState: .collectionState,
  lastCollected: .lastCollected
}'

NSX Adapter

ops_api "/api/adapters?adapterKindKey=NSXTAdapter" | jq '.adapterInstancesInfoDto[] | {
  name: .resourceKey.name,
  collectionState: .collectionState
}'

vSAN Adapter

ops_api "/api/adapters?adapterKindKey=VsanAdapter" | jq '.adapterInstancesInfoDto[] | {
  name: .resourceKey.name,
  collectionState: .collectionState
}'

SDDC Health Adapter

ops_api "/api/adapters?adapterKindKey=SDDCHealthAdapter" | jq '.adapterInstancesInfoDto[] | {
  name: .resourceKey.name,
  collectionState: .collectionState
}'

15. API Health (Suite API)

Token Acquisition Test

time curl -sk -X POST \
  "https://$OPS/suite-api/api/auth/token/acquire" \
  -H "Content-Type: application/json" \
  -d "{\"username\":\"$OPS_USER\",\"password\":\"$OPS_PASS\",\"authSource\":\"local\"}" \
  | jq -r '.token' > /dev/null

Endpoint Responsiveness

ENDPOINTS="/api/deployment/node /api/adapters /api/resources?pageSize=1 /api/alerts?pageSize=1"
for EP in $ENDPOINTS; do
  START=$(date +%s%N)
  HTTP=$(curl -sk -o /dev/null -w "%{http_code}" \
    -H "Authorization: vRealizeOpsToken $OPS_TOKEN" \
    "https://$OPS/suite-api$EP")
  END=$(date +%s%N)
  MS=$(( (END - START) / 1000000 ))
  echo "$EP: HTTP $HTTP (${MS}ms)"
done

16. NTP & DNS

ssh root@$OPS
# NTP
timedatectl status
chronyc tracking

# DNS
nslookup vcenter.lab.local
nslookup nsx-vip.lab.local
cat /etc/resolv.conf

17. Backup Configuration

# Check via CASA
curl -sk "https://$OPS/casa/deployment/backup/schedule" \
  -u "admin:$OPS_PASS" | jq .

18. Resource Utilization

ssh root@$OPS
# CPU and Load
uptime
top -b -n 1 | head -5

# Memory
free -m

# Disk
df -h

# Java heap (analytics process)
ps aux | grep analytics | grep -v grep | awk '{print $6/1024 " MB"}'

19. Port Reference Table

Inbound Ports

Outbound Ports

Inter-Node Ports

20. Common Issues & Remediation

20.1 Cluster Offline

20.2 Slice Degraded

20.3 Adapter Failures

20.4 Disk Full

# Quick disk cleanup
ssh root@$OPS
# 1. Clean old logs
find /storage/log -name "*.gz" -mtime +14 -delete
# 2. Check large files
du -sh /storage/* | sort -rh | head -10
# 3. Compact Cassandra
/opt/vmware/vcops/cassandra/apache-cassandra/bin/nodetool compact

20.5 Collection Gaps

20.6 Certificate Expiry

21. CLI Quick Reference Card

CASA Admin CLI

System Commands

Cassandra Commands

Log Locations

22. API Quick Reference (Suite API)

Authentication

# Acquire token
curl -sk -X POST "https://$OPS/suite-api/api/auth/token/acquire" \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"password","authSource":"local"}'

# Release token
curl -sk -X POST "https://$OPS/suite-api/api/auth/token/release" \
  -H "Authorization: vRealizeOpsToken $OPS_TOKEN" \
  -H "Content-Type: application/json"

Key Endpoints

Common Query Parameters