VMware Cloud Foundation 9.0

PROFESSIONAL DEPLOYMENT & TROUBLESHOOTING HANDBOOK

Document Version: 3.0 Professional Edition Based on: VMware Cloud Foundation 9.0 / 9.0.1 Official Documentation Verified Against: Broadcom TechDocs (January 2026) Certification Alignment: VCF Administrator Exam (2V0-17.25) Total Content: 400+ Detailed Procedures, 150+ Study Questions

TABLE OF CONTENTS

Part I: VCF 9.0 Overview & Architecture

Part II: Deployment & Configuration

Part III: Core Component Management

Part IV: VCF Operations & Automation

Part V: Troubleshooting Guide

Part VI: Quick Reference

Appendices

PART I: VCF 9.0 OVERVIEW & ARCHITECTURE

Section 1: What's New in VCF 9.0

1.1 Executive Summary

VMware Cloud Foundation 9.0 represents a major architectural evolution of the VMware private cloud platform. This release introduces significant changes to deployment, licensing, and operations that administrators must understand before deployment or upgrade.

Key Transformation Points

1.2 New Features & Capabilities

Unified Operations Interface

VCF 9.0 provides a streamlined experience for building, operating, and securing private cloud from a single interface through VCF Operations.

Key Benefits: - Single pane of glass for operations - Integrated governance and compliance - Fleet management across multiple VCF instances - Proactive health monitoring and recommendations

Self-Service Cloud Consumption

VCF Automation enables self-service private cloud with built-in services: - Virtual Machines - Kubernetes clusters - Networking (VPC) - Databases - Container registries - AI workloads (Private AI Foundation)

Integrated Container & VM Platform

Kubernetes and virtualization are integrated out of the box: - No separate stack assembly required - Developers can deploy workloads immediately - Unified lifecycle management - Consistent networking across VMs and containers

Security & Compliance (FIPS Mode)

All VCF 9.0 components are updated with NIST recommended standards: - FIPS 140-2 and 140-3 compliant - vCenter, ESX, and NSX run in FIPS-enabled mode by default - This mode cannot be deactivated - Critical for government and regulated industry deployments

1.3 Deprecated & Removed Features

SDDC Manager UI Deprecation

• The standalone SDDC Manager UI is deprecated
• Will be removed in a future major release
• Functionality moving to VCF Operations Fleet Management
• SDDC Manager remains as backend component for API operations

vSphere Lifecycle Manager Baselines

• Managing clusters with baselines and baseline groups is no longer supported
• Must use vLCM Images (desired state configuration)
• Existing environments must migrate before upgrade

Integrated Windows Authentication (IWA)

• vCenter 9.0 discontinues IWA support
• Migrate to Active Directory over LDAPS
• Or use Identity Federation with Multi-Factor Authentication

Host Profiles

• vSphere Host Profiles deprecated in vCenter 9.0
• Will be removed in future release
• Use vSphere Configuration Profiles instead (cluster-level)

1.4 Component Versions (Bill of Materials)

VCF 9.0 Core Components

VCF 9.0.1 Maintenance Release

• Resolves CVE-2025-41251 and CVE-2025-41252
• Security and bug fixes
• Hardware enablement updates
• Backward-compatible new features

Section 2: Core Architecture & Components

2.1 VCF Architecture Overview

VMware Cloud Foundation is an integrated software stack that bundles:

┌─────────────────────────────────────────────────────────────────┐
│                    VCF OPERATIONS (Mandatory)                    │
│         Fleet Management | Monitoring | Diagnostics              │
├─────────────────────────────────────────────────────────────────┤
│                     VCF AUTOMATION (Optional)                    │
│      Self-Service | Blueprints | Service Broker | Orchestrator   │
├─────────────────────────────────────────────────────────────────┤
│                       SDDC MANAGER                               │
│       Lifecycle Management | Deployment | Orchestration          │
├────────────────┬────────────────┬────────────────┬──────────────┤
│   vSphere      │      NSX       │     vSAN       │   vCenter    │
│   (Compute)    │  (Networking)  │   (Storage)    │  (Mgmt)      │
├────────────────┴────────────────┴────────────────┴──────────────┤
│                     ESXi HYPERVISOR                              │
│                   Type 1 Bare-Metal                              │
└─────────────────────────────────────────────────────────────────┘

2.2 Core Components Detailed

SDDC Manager

Purpose: Central lifecycle management and orchestration platform

Key Functions: - Automated deployment of workload domains - Patching and upgrades across the stack - Certificate management - Password rotation - Health monitoring integration - REST API for automation

Key Services: | Service | Purpose | |---------|---------| | domainmanager | Domain lifecycle operations | | lcm | Lifecycle management | | operationsmanager | Operations and monitoring | | commonsvcs | Shared platform services | | postgresql | Internal database | | nginx | Web server/reverse proxy |

Log Location: /var/log/vmware/vcf/

vCenter Server

Purpose: Compute virtualization management

Key Functions: - ESXi host management - VM lifecycle operations - DRS (Distributed Resource Scheduler) - HA (High Availability) - vMotion orchestration

Key Services: | Service | Purpose | |---------|---------| | vpxd | Core vCenter daemon | | vsphere-ui | vSphere Client | | vmware-postgres | Embedded database | | sso | Single Sign-On | | vlcm | vSphere Lifecycle Manager |

Log Location: /var/log/vmware/

NSX Manager

Purpose: Software-defined networking and security

Key Functions: - Overlay networking (GENEVE) - Micro-segmentation (DFW) - Gateway firewalls - Load balancing - VPN services

Architecture: | Component | Purpose | |-----------|---------| | NSX Manager Cluster | 3-node management/control plane | | Transport Zones | Define segment scope | | Segments | Layer 2 logical networks | | Tier-0 Gateway | North-south routing | | Tier-1 Gateway | Tenant/application routing | | TEP (Tunnel Endpoint) | Overlay encapsulation | | DFW | Distributed Firewall |

MTU Requirement: Minimum 1600 bytes for overlay traffic

vSAN

Purpose: Software-defined storage

Architectures: | Type | Description | |------|-------------| | vSAN OSA | Original Storage Architecture - disk groups with cache/capacity tiers | | vSAN ESA | Express Storage Architecture - single tier, NVMe only |

Key Concepts: - FTT (Failures to Tolerate): Data protection level - RAID Policies: RAID-1 (mirror) or RAID-5/6 (erasure coding) - Storage Policies: Define protection for VMs - Disk Groups: Cache + capacity disks (OSA only)

Host Requirements by FTT: | FTT | RAID-1 | RAID-5/6 | |-----|--------|----------| | 1 | 3 hosts | 4 hosts | | 2 | 5 hosts | 6 hosts | | 3 | 7 hosts | N/A |

2.3 Workload Domains

Management Domain

• Created during initial VCF deployment
• Hosts ALL infrastructure components
• Minimum 4 ESXi hosts
• Should NOT run production workloads
• Contains: SDDC Manager, vCenter, NSX Managers, VCF Operations

VI Workload Domains

• Created after initial deployment for customer workloads
• Has dedicated or shared vCenter
• Shares NSX infrastructure with Management Domain
• Minimum 3 ESXi hosts
• Can be added/removed as needed
• Isolates workloads from management infrastructure

Consolidated vs Standard Architecture

Section 3: Licensing & Entitlements

3.1 VCF 9.0 Licensing Model

Simplified Licensing Structure

VCF 9.0 reduces complexity from 11 license keys to just 2:

Per-Core Licensing Details

• 16-core minimum per CPU
• Must license ALL physical cores on each host
• Including any BIOS-disabled cores
• Significant change from previous per-socket model

License Management

• Multiple subscriptions pool into a single license
• Single license file imports all entitlements
• Can optionally split licenses later
• 90-day evaluation mode available

Connected Mode Requirements

• License usage submissions required every 90 days
• Automated through VCF Operations connected to Broadcom Support Portal
• Maintains entitlement validation

3.2 VCF Licensing Tiers

Note: VCF Operations is mandatory across all tiers in VCF 9.0.

PART II: DEPLOYMENT & CONFIGURATION

Section 4: VCF Installer & Initial Deployment

4.1 VCF Installer Overview

The VCF Installer is a new appliance in VCF 9.0 that replaces Cloud Builder.

Key Differences from Cloud Builder:

4.2 Pre-Deployment Requirements

Network Prerequisites

┌─────────────────────────────────────────────────────────────────┐
│ NETWORK REQUIREMENTS CHECKLIST                                   │
├─────────────────────────────────────────────────────────────────┤
│ □ Management VLAN configured on all ToR switches                │
│ □ vMotion VLAN configured (if separate)                         │
│ □ vSAN VLAN configured (if separate)                            │
│ □ TEP VLAN for NSX overlay (MTU 1600+ required)                 │
│ □ DNS forward and reverse records for all components            │
│ □ NTP server accessible from management network                 │
│ □ Default gateway reachable from all hosts                      │
└─────────────────────────────────────────────────────────────────┘

Host Prerequisites

┌─────────────────────────────────────────────────────────────────┐
│ HOST REQUIREMENTS CHECKLIST                                      │
├─────────────────────────────────────────────────────────────────┤
│ □ Minimum 4 hosts for Management Domain                         │
│ □ ESXi installed from VMware ISO (clean state)                  │
│ □ Only vSwitch0 with vmk0 for management                        │
│ □ No existing vCenter connections                               │
│ □ DNS and NTP configured correctly                              │
│ □ Hardware on VMware HCL                                        │
│ □ Sufficient CPU, memory, and storage                           │
└─────────────────────────────────────────────────────────────────┘

4.3 Deployment Phases

Phase 1: Bring-up (VCF Installer)

1. Deploy VCF Installer OVA
2. Access web UI
3. Upload deployment parameters JSON
4. VCF Installer validates prerequisites
5. Deploys: ESXi config, vCenter, SDDC Manager, NSX, vSAN
6. Creates Management Domain

Phase 2: Workload Domain Creation (SDDC Manager)

1. Commission additional ESXi hosts
2. Create Workload Domain via SDDC Manager
3. Domain gets dedicated resources
4. Optional: Deploy workload vCenter

Phase 3: Day-2 Operations (VCF Operations)

1. Ongoing lifecycle management
2. Patching and upgrades
3. Monitoring and health checks
4. Capacity planning

4.4 Air-Gapped Deployment

For environments without internet access:

Using VCF Download Tool

# Step 1: On internet-connected machine
./vcf-download-tool --product VCF --version 9.0

# Step 2: Transfer to SDDC Manager
scp vcf-bundle-*.tar admin@<sddc-manager>:/tmp/

# Step 3: Upload via SDDC Manager UI
# Navigate to: Lifecycle Management → Bundle Management → Upload Bundle

Required Binaries for Air-Gapped Deployment

Section 5: Workload Domains

5.1 Creating Workload Domains

Prerequisites Before Creation

┌─────────────────────────────────────────────────────────────────┐
│ WORKLOAD DOMAIN PREREQUISITES                                    │
├─────────────────────────────────────────────────────────────────┤
│ □ ESXi hosts installed with supported version                   │
│ □ Hosts commissioned into VCF from VCF Installer                │
│ □ Network connectivity validated                                │
│ □ Storage prepared (vSAN, FC, NFS)                              │
│ □ License entitlements available                                │
└─────────────────────────────────────────────────────────────────┘

Storage Options for Workload Domains

Principal Storage (Primary): | Type | Management Domain | Workload Domains | |------|-------------------|------------------| | vSAN OSA | ✓ | ✓ | | vSAN ESA | ✓ | ✓ | | VMFS-FC | ✓ | ✓ | | NFSv3 | ✓ | ✓ | | NVMe/TCP | ✗ | ✓ | | vVols | ✗ | ✓ |

Fibre Channel Prerequisites

• Storage LUNs must be presented to ALL ESXi hosts
• VMFS datastore must be mounted on all hosts
• Zoning and masking configured in storage fabric

5.2 Workload Domain with Dedicated NSX

When to use dedicated NSX instance: - Complete isolation requirements - Independent scaling needed - Separate security policies - Different lifecycle management

# Deploy new cluster as separate workload domain with new NSX instance
# This ensures lifecycle and scaling are fully isolated

Section 6: Identity & Security Configuration

6.1 VCF Single Sign-On

SSO Configuration Steps

1. Configure vCenter SSO Identity Sources
2. Add Active Directory as identity source
3. Options: AD over LDAPS or Identity Federation with MFA
4. Note: IWA is discontinued in vCenter 9.0

Service Roles Required for RBAC

After SSO configuration, assign roles on these components: | Component | Purpose | |-----------|---------| | VMware NSX Manager | Networking and security services | | VMware vCenter | Compute and cluster administration | | VCF Operations | Monitoring and lifecycle capabilities |

6.2 Certificate Management

Certificate Format Requirements

• Required Format: PEM (Base64-encoded)
• Contains certificate and key
• Standard for all VCF components

Certificate Replacement Procedure

1. Generate CSR from component or externally
2. Submit to Certificate Authority
3. Receive signed certificate in PEM format
4. Import via SDDC Manager or component UI
5. Verify certificate chain is complete

6.3 Password Management

Password Functions in VCF

Best Practices

• Always change passwords through VCF Operations
• Set up automatic rotation (30, 60, or 90 days)
• If emergency change needed outside VCF, immediately use Update function

PART III: CORE COMPONENT MANAGEMENT

Section 7: SDDC Manager Operations

7.1 SDDC Manager Services

Service Status Commands

# Check all VCF services
systemctl status vcf-services

# Check individual services
systemctl status domainmanager
systemctl status lcm
systemctl status operationsmanager
systemctl status nginx
systemctl status postgresql

# Restart all services
systemctl restart vcf-services

# Restart individual service
systemctl restart domainmanager

Log File Locations

/var/log/vmware/vcf/
├── domainmanager/
│   └── domainmanager.log
├── lcm/
│   └── lcm.log
├── operationsmanager/
│   └── operationsmanager.log
├── commonsvcs/
│   └── commonsvcs.log
└── sddc-support/
    └── sddc-support.log

7.2 Bundle Management

Online Bundle Download

1. Navigate to Lifecycle Management → Bundle Management
2. Click "Download Now" for desired bundle
3. Monitor progress (30-60 minutes for large bundles)
4. Bundles require 10-50GB depending on content

Offline Bundle Upload

# Transfer bundle to SDDC Manager
scp vcf-bundle-*.tar admin@<sddc-manager>:/tmp/

# Then upload via UI:
# Lifecycle Management → Bundle Management → Upload Bundle

Troubleshooting Bundle Download Failures

1. Check internet connectivity: curl -I https://depot.vmware.com
2. Verify proxy settings in Administration → Network Settings
3. Verify Broadcom credentials in Administration → Depot Settings
4. Check disk space: df -h /

Section 8: vCenter Server Management

8.1 vCenter Services

Service Management Commands

# Check all services status
service-control --status --all

# Restart specific service
service-control --restart vpxd
service-control --restart vsphere-ui

# Restart all services (causes brief outage)
service-control --restart --all

# Wait 10-15 minutes for all services to start

Critical vCenter Services

Log Locations

/var/log/vmware/
├── vpxd/
│   └── vpxd.log
├── vsphere-ui/
│   └── logs/vsphere_client_virgo.log
├── sso/
│   └── vmware-sts-idmd.log
└── vpostgres/
    └── postgresql-*.log

8.2 Enhanced vMotion Compatibility (EVC)

When EVC is Required

• Cluster has hosts with different CPU generations
• Need to enable DRS automatic VM migration
• Mixed Intel or AMD processor types

EVC Mode Selection

Choose the mode matching your OLDEST CPU generation:

Intel EVC Hierarchy (newest to oldest):
├── Cascade Lake
├── Skylake
├── Broadwell
├── Haswell
├── Ivy Bridge
└── Sandy Bridge

Enabling EVC

1. Power off all VMs in cluster (or ensure already at lower EVC)
2. Right-click cluster → Settings → VMware EVC
3. Select appropriate mode (oldest CPU generation)
4. Power on VMs - they inherit cluster EVC mode

Section 9: NSX Networking & Security

9.1 NSX Architecture in VCF

Component Hierarchy

┌─────────────────────────────────────────────────────────────────┐
│                    NSX MANAGER CLUSTER                          │
│                    (3-node for HA)                               │
├─────────────────────────────────────────────────────────────────┤
│                      TIER-0 GATEWAY                              │
│              (Provider Router - North-South)                     │
│                    BGP/OSPF to Physical                          │
├─────────────────────────────────────────────────────────────────┤
│                      TIER-1 GATEWAY                              │
│              (Tenant Router - Internal)                          │
│                   NAT, Load Balancing                            │
├─────────────────────────────────────────────────────────────────┤
│                        SEGMENTS                                  │
│              (Layer 2 - Overlay or VLAN)                         │
└─────────────────────────────────────────────────────────────────┘

Transport Zones

TEP (Tunnel Endpoint) Requirements

• Dedicated VLAN for TEP traffic
• MTU must be 1600+ bytes (GENEVE adds ~54 bytes overhead)
• Can use DHCP or static IP pool
• Typically 2 TEP interfaces for redundancy

9.2 Creating Segments

VLAN-Backed Segment Requirements

1. VLAN ID - Maps to physical network
2. VLAN Transport Zone - Must be VLAN type

NOT required for VLAN segments: - Tier-1 gateway connection - Subnet gateway IP - DHCP configuration (physical network handles)

Overlay Segment Requirements

1. Overlay Transport Zone
2. Connected to Tier-1 Gateway (for routing)
3. Subnet and gateway configuration
4. Optional DHCP configuration

9.3 VKS Network Connectivity

For VMware Kubernetes Service clusters backed by NSX: - Centralized Connectivity is required - Provides routed access through NSX Tier-0/Tier-1 gateways - Enables external access to Kubernetes services

9.4 Distributed Firewall (DFW)

Rule Processing Order (Highest to Lowest Priority)

1. Emergency - Critical security policies
2. Infrastructure - Protect infrastructure components
3. Environment - Zone-based policies
4. Application - App-specific micro-segmentation
5. Default - Catch-all rules

NSX Credentials Managed by VCF

Section 10: vSAN Storage Management

10.1 vSAN Architectures

vSAN OSA (Original Storage Architecture)

• Uses disk groups
• Requires separate cache and capacity tiers
• Cache tier: 1 SSD per disk group
• Capacity: Up to 7 disks per group
• Max 5 disk groups per host
• Recommended for: Nested/lab environments, existing infrastructure

vSAN ESA (Express Storage Architecture)

• No disk groups - single tier
• NVMe SSDs only (high-performance)
• Better performance and efficiency
• Higher compression/deduplication
• Minimum: 4 NVMe devices per host
• Recommended for: New VCF 9.0 deployments

10.2 Storage Policies

FTT (Failures to Tolerate)

vSAN ESA with Auto-Policy Management

• RAID-6 with auto-policy requires minimum 8 hosts

10.3 vSAN File Services

Provides NFS/SMB file shares accessible to both VMs and external clients.

Use Case: Storage accessible to both client workstations and VMs

Section 11: ESXi Host Management

11.1 Host Commissioning

Prerequisites for Commissioning

1. ESXi installed from supported VMware ISO
2. Clean state - no existing vCenter connection
3. Only vSwitch0 with vmk0 for management
4. DNS resolution working (forward and reverse)
5. NTP synchronized
6. Compatible hardware per VMware HCL

Commissioning Process

1. Pre-install supported ESXi version using valid ISO
2. Commission hosts into VCF from VCF Installer
3. VCF validates and onboards hosts
4. Hosts ready for workload domain deployment

11.2 SSL Certificate Regeneration

# Regenerate SSL certificates on ESXi host
/sbin/generate-certificates

11.3 Persisting Configuration Changes

# After network configuration changes, persist across reboots
/sbin/auto-backup.sh

PART IV: VCF OPERATIONS & AUTOMATION

Section 12: VCF Operations Monitoring

12.1 VCF Operations Deployment Models

Scaling Limitation: Simple model cannot be upgraded in place - must redeploy as HA.

12.2 Monitoring Capabilities

VCF Health Dashboard

Monitors three core components: - vCenter Server - NSX - ESX hosts

VCF Diagnostics

• Run diagnostics for known problems
• Security vulnerability detection
• Remediation recommendations

12.3 Super Metrics

Purpose: Aggregate metrics across objects (e.g., average VM health across clusters)

Creation Steps: 1. Navigate to Administration → Configuration → Super Metrics 2. Define formula using available metrics 3. Enable in Active Policy

12.4 Alert Configuration

Creating Alerts

1. Create Symptom Definition (condition to monitor)
2. Create Alert Definition (links symptoms to alert)
3. Enable the alert in an Active Policy (required!)

Wait Cycle and Cancel Cycle

• Default cycle: 20 minutes
• Wait cycle of 2 = 40 minutes before alert triggers
• Cancel cycle of 2 = 40 minutes before alert auto-clears

12.5 Configuration Drift Detection

Prerequisites

• vCenter version 8.0 U3 or later
• vSphere Configuration Profiles for each cluster
• Scheduled Drift Detection in VCF Operations

Section 13: VCF Automation

13.1 VCF Automation Components

13.2 Organization Types

Recommendation: Use "All Applications" for new organizations (required for Tanzu Salt, AD integrations, NPC endpoints)

13.3 Provider Networking Prerequisite

Must complete before configuring Provider Networking: - Create a Tier-0 Gateway in NSX Manager

13.4 Regional Networking (All Applications)

When creating regional networking, these NSX constructs are automatically configured: 1. Outbound SNAT rule 2. Default VPC 3. Provider Tier-0 Gateway 4. VPC connectivity profile

13.5 Event Subscriptions and Extensibility

Automatic CMDB Registration Example

Use these components for auto-registering Kubernetes deployments: 1. Event Subscriptions - Capture deployment lifecycle events 2. Action-based Extensibility (ABX) - Execute API calls to CMDB

Section 14: Kubernetes & Container Services

14.1 vSphere Supervisor

The Supervisor Cluster is the vSphere control plane integration with Kubernetes: - Enables Kubernetes capabilities on vSphere - Foundation for TKG guest/workload clusters - Runs on ESXi hosts

14.2 VMware Kubernetes Service (VKS)

Default CNI

Antrea is the default Container Network Interface for VKS workload clusters.

Cluster Provisioning

Cluster API is used to provision and manage Kubernetes workload clusters.

14.3 Container Storage Interface (CSI)

Purpose: Allows storage providers to expose storage as persistent volumes for Kubernetes workloads.

14.4 VM Classes in vSphere Supervisor

VM Classes define compute characteristics: - CPU resources - Memory resources

14.5 vSphere Namespace Resource Limits

Configurable limits: - CPU - Memory - Storage

14.6 Key Kubernetes Components

PART V: TROUBLESHOOTING GUIDE

Section 15: SDDC Manager Troubleshooting

15.1 UI Inaccessible

Symptom

• Cannot access https://
• Connection timeout or refused

Resolution Steps

Step 1: Verify Network Connectivity

ping <sddc-manager-ip>

Step 2: Verify VM is Running - Check in vCenter that SDDC Manager VM is powered on

Step 3: SSH to Appliance

ssh admin@<sddc-manager-ip>

Step 4: Check Services

systemctl status vcf-services
systemctl status domainmanager
systemctl status nginx

Step 5: Review Logs

tail -100 /var/log/vmware/vcf/domainmanager/domainmanager.log
grep -i error /var/log/nginx/error.log

Step 6: Restart Services

systemctl restart vcf-services

Step 7: Check Database

systemctl status postgresql
sudo -u postgres psql -c "SELECT 1;"

15.2 Task Failures

Diagnosis Steps

1. Identify Failed Task
2. Navigate to Inventory → Tasks

3. Filter by Status = "Failed"

4. Analyze Subtasks

5. Expand failed task
6. Identify which subtask failed

7. Read error message

8. Check LCM Logs bash grep "<task-id>" /var/log/vmware/vcf/lcm/lcm.log

Common Issues and Resolutions

Clearing Stuck Tasks (Advanced)

# Get authentication token
curl -k -X POST https://localhost/v1/tokens \
  -H "Content-Type: application/json" \
  -d '{"username":"admin@local","password":"<password>"}'

# Cancel stuck task
curl -k -X PATCH https://localhost/v1/tasks/<task-id> \
  -H "Authorization: Bearer <access-token>" \
  -H "Content-Type: application/json" \
  -d '{"status":"CANCELLED"}'

Section 16: vCenter Troubleshooting

16.1 Services Not Running

Resolution Steps

Step 1: Check All Services

service-control --status --all

Step 2: Restart Failed Services

service-control --restart <service-name>

Step 3: Check Logs

tail -100 /var/log/vmware/vpxd/vpxd.log
grep -i "error\|exception" /var/log/vmware/vpxd/vpxd.log

Step 4: Check Database

service-control --status vmware-vpostgres
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"

16.2 Password Out of Sync with VCF

Resolution

1. Navigate to Administration → Password Management in SDDC Manager
2. Locate the credential (e.g., vcenter-root)
3. Click menu → "Update Password"
4. Enter the NEW password (currently set in vCenter)
5. Verify status shows "In Sync"

Section 17: NSX Troubleshooting

17.1 Transport Node Connectivity

Diagnosis Steps

Step 1: Check Status in NSX Manager - Navigate to System → Fabric → Nodes → Host Transport Nodes - Review status (green/yellow/red)

Step 2: Test TEP Connectivity

# SSH to ESXi host
ssh root@<esxi-host-ip>

# Find TEP VMkernel
esxcfg-vmknic -l | grep -i tep

# Test basic connectivity
vmkping <other-host-tep-ip>

# Test with jumbo frames (MTU 1600)
vmkping -d -s 1572 <other-host-tep-ip>

Step 3: Check NSX Agent

/etc/init.d/nsx-proxy status
/etc/init.d/nsx-datapath status
cat /var/log/nsx-syslog.log | tail -50

Step 4: Resync Transport Node - NSX Manager → System → Fabric → Nodes - Click problematic host → Actions → Redeploy Node

17.2 Traceflow for Network Debugging

Steps

1. NSX Manager → Plan & Troubleshoot → Traffic Analysis → Traceflow
2. Select Source (VM or IP)
3. Select Destination (VM or IP)
4. Select Protocol (ICMP, TCP, UDP)
5. Click "Trace"

Interpreting Results

Section 18: vSAN Troubleshooting

18.1 Objects Non-Compliant

Common Causes

• Insufficient hosts for FTT policy
• Insufficient free disk space
• Host/disk failure
• Resyncing in progress

Resolution Steps

1. Check vSAN Health

2. vCenter → Cluster → Monitor → vSAN → Health

3. Check Object Compliance

4. Monitor → vSAN → Virtual Objects

5. Review compliance status details

6. CLI Investigation bash esxcli vsan health cluster list esxcli vsan storage list

7. Resolution Options

8. If host/disk failed: Replace and wait for rebuild
9. If policy issue: Modify policy or add capacity
10. If network: Fix vSAN VMkernel connectivity

18.2 Nested ESXi vSAN Issues

Disks Showing as HDD

Cause: Virtual SATA disks not advertised as SSD

Fix: 1. Add to VMX file: sata0:0.virtualSSD = 1 2. Full power cycle required (not just reboot)

Stale vSAN Metadata

Symptom: "Used by this host: false" in vSAN storage list

Fix:

# Unmount disk group
esxcli vsan storage diskgroup unmount -s <ssd> -d <capacity>

# Wipe metadata (recreate GPT)
partedUtil mklabel /dev/disks/<device> gpt

Section 19: Lifecycle Management & Upgrades

19.1 VCF Upgrade Order

Critical: Follow this order exactly: 1. SDDC Manager - Always first 2. vCenter Server 3. NSX Manager cluster 4. ESXi Hosts (rolling upgrade) 5. vSAN 6. VCF Operations

Why Order Matters

• SDDC Manager must understand new versions first
• Backward compatibility is designed top-down
• Newer vCenter can manage older ESXi (not vice versa)

19.2 Upgrade Prerequisites

Pre-Upgrade Checklist

┌─────────────────────────────────────────────────────────────────┐
│ BEFORE UPGRADING                                                 │
├─────────────────────────────────────────────────────────────────┤
│ □ Check VCF Compatibility Matrix                                │
│ □ Download required bundles                                     │
│ □ Take backups/snapshots of management components               │
│ □ Verify VCF Health shows all green                             │
│ □ Schedule maintenance window                                   │
│ □ Notify stakeholders                                           │
│ □ Document current configuration                                │
└─────────────────────────────────────────────────────────────────┘

19.3 Converging Existing vSphere to VCF

Prerequisites

• vCenter Server VM must be hosted on cluster it manages
• Lifecycle Manager configured with Images only (not baselines)
• Enhanced Linked Mode must be deactivated
• vSphere Distributed Switch version 7.0 or later

Required Components for Upgrade to VCF 9.0

• VCF Operations
• VCF Operations Fleet Management
• VCF Identity Broker

PART VI: QUICK REFERENCE

Section 20: Command Reference

SDDC Manager Commands

# Service management
systemctl status vcf-services
systemctl restart vcf-services
systemctl status domainmanager
systemctl restart domainmanager

# Log viewing
tail -f /var/log/vmware/vcf/domainmanager/domainmanager.log
grep -i error /var/log/vmware/vcf/lcm/lcm.log

vCenter Commands

# Service management
service-control --status --all
service-control --restart --all
service-control --restart vpxd

# Database check
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT 1;"

ESXi Commands

# Network information
esxcfg-vmknic -l
esxcli network ip interface list

# vSAN commands
esxcli vsan health cluster list
esxcli vsan storage list
esxcli vsan storage diskgroup unmount -s <ssd> -d <capacity>

# Certificate regeneration
/sbin/generate-certificates

# Persist configuration
/sbin/auto-backup.sh

# Disk operations
partedUtil mklabel /dev/disks/<device> gpt

NSX Commands (on ESXi)

# Agent status
/etc/init.d/nsx-proxy status
/etc/init.d/nsx-datapath status

# Connectivity test
vmkping <tep-ip>
vmkping -d -s 1572 <tep-ip>  # Jumbo frame test

Section 21: Port Requirements

SDDC Manager Ports

vCenter Ports

NSX Manager Ports

vSAN Ports

Section 22: Log File Locations

SDDC Manager

vCenter

ESXi

APPENDICES

Appendix A: Common Error Messages & Resolutions

Appendix B: Exam Study Questions by Topic

VCF Automation Key Points

• VCF Service Broker provides self-service catalog in VM Apps tenant
• Provider Networking requires Tier-0 Gateway in NSX Manager (prerequisite)
• VCF Automation Simple model cannot scale up in place - must redeploy
• Content Sharing Policies control catalog visibility across tenants
• Event Subscriptions + ABX enable automatic CMDB registration
• All Applications organization type required for Tanzu Salt, AD integrations, NPC

VCF Operations Key Points

• Simple deployment has smallest footprint
• Update Password function syncs password changed outside VCF
• Alerts must be enabled in Active Policy to trigger
• Super metrics aggregate values across objects
• vCenter 8.0 U3+ required for configuration drift detection
• VCF Operations for Networks enables Crown Jewel Analysis

NSX Key Points

• VLAN-backed segment requires: VLAN ID + VLAN Transport Zone
• Centralized Connectivity required for VKS clusters
• MTU 1600+ bytes required for overlay traffic (GENEVE overhead)
• DFW rule order: Emergency > Infrastructure > Environment > Application > Default
• VCF manages NSX credentials: admin, audit, backup

vSAN Key Points

• Storage Overview shows all vSAN and non-vSAN datastores
• vSAN ESA RAID-6 with auto-policy requires minimum 8 hosts
• Witness Host provides quorum in stretched clusters
• vSAN File Services provides NFS/SMB to both VMs and external clients
• vSAN OSA recommended for nested/lab environments

Kubernetes Key Points

• Antrea is default CNI for VKS workload clusters
• Cluster API provisions workload clusters
• CSI provides persistent storage to containers
• VM Classes define CPU and memory for VMs
• Velero backs up and restores Kubernetes clusters
• Supervisor Cluster enables Kubernetes on vSphere

Lifecycle & Deployment Key Points

• VCF Installer replaces Cloud Builder in VCF 9.0
• ESXi and NSX binaries required for air-gapped deployment
• VMFS datastore must be mounted on all hosts before commissioning (FC storage)
• SDDC Manager upgraded first before other components
• vLCM Images replace baselines in vCenter 9.0
• PEM format required for external CA certificates

Document Information

Version History: | Version | Date | Changes | |---------|------|---------| | 1.0 | Jan 2026 | Initial release | | 2.0 | Jan 2026 | Added troubleshooting procedures | | 3.0 | Jan 2026 | Verified against Broadcom TechDocs, added exam content |

Sources: - Broadcom TechDocs: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html - VCF 9.0 Release Notes: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/release-notes/vmware-cloud-foundation-90-release-notes.html - VCF 9.0.1 Release Notes: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/release-notes/vmware-cloud-foundation-9-0-1-release-notes.html

This handbook is intended for VMware Cloud Foundation administrators and engineers preparing for deployment, operations, and certification.

ADDITIONAL APPENDICES

Appendix C: Cloud Builder Offline Depot Setup

C.1 Python HTTPS Depot Server Configuration

Complete Server Setup

Required Components: 1. Python 3.x with ssl module 2. ThreadingMixIn for concurrent requests 3. Self-signed SSL certificate with SAN 4. Proper cipher configuration

Certificate Generation

# Set variable to prevent path mangling in Git Bash (Windows)
export MSYS_NO_PATHCONV=1

# Generate 4096-bit RSA key and self-signed certificate
openssl req -x509 \
  -newkey rsa:4096 \
  -keyout depot-key.pem \
  -out depot-cert.pem \
  -days 365 \
  -nodes \
  -subj '/CN=depot-server' \
  -addext "subjectAltName=IP:192.168.1.100,IP:10.0.0.100"

Server Configuration

# Key server settings
context.minimum_version = ssl.TLSVersion.TLSv1_2
context.options |= ssl.OP_NO_RENEGOTIATION  # Prevent Java TLS issues
server.protocol_version = 'HTTP/1.1'  # Required for Java clients
context.set_ciphers('DEFAULT:!aNULL:!MD5:!DSS')  # Compatible ciphers

Common Issues and Solutions

C.2 Cloud Builder Certificate Import

Steps to Import Certificate

# 1. Download certificate from depot server
openssl s_client -connect <depot-ip>:8443 </dev/null 2>/dev/null | \
  openssl x509 > /tmp/depot.crt

# 2. Import into Java truststore
keytool -import -alias vcf-depot \
  -file /tmp/depot.crt \
  -keystore /usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts \
  -storepass changeit \
  -noprompt

# 3. Restart Cloud Builder services
systemctl restart commonsvcs domainmanager lcm operationsmanager

Service Restart Order

After certificate import, restart these services: 1. commonsvcs (Platform Services) 2. domainmanager (Domain Manager) 3. lcm (Lifecycle Management) 4. operationsmanager (Operations Manager)

Note: 502 errors for 2-3 minutes after restart are normal.

C.3 Offline Depot Directory Structure

DEPOT_ROOT/
├── PROD/
│   ├── metadata/
│   │   └── productVersionCatalog.json  # Master manifest
│   ├── vsan/
│   │   └── hcl/
│   │       ├── all.json
│   │       └── lastupdatedtime.json
│   └── COMP/
│       ├── VCENTER/
│       │   ├── VMware-VCSA-*.iso
│       │   └── VMware-vCenter-*-updaterepo.zip
│       ├── NSX_T_MANAGER/
│       │   └── nsx-*.ova
│       ├── SDDC_MANAGER_VCF/
│       │   └── sddc-manager-*.ova
│       └── ESX_HOST/
│           └── VMware-ESXi-*.zip

Appendix D: Pre-Flight Checklists

D.1 Management Domain Deployment Checklist

Network Verification

□ Management VLAN configured on ToR switches
□ vMotion VLAN configured (MTU 9000 recommended)
□ vSAN VLAN configured (MTU 9000 recommended)
□ TEP VLAN configured (MTU 1600+ REQUIRED)
□ DNS A records created for all components
□ DNS PTR records created for all components
□ NTP server accessible and synchronized
□ Default gateway responds to ping
□ Upstream switches support required MTUs

Host Verification (Each ESXi Host)

□ ESXi installed from VMware ISO
□ Only vSwitch0 with vmk0 exists
□ No existing vCenter connection
□ Root password set and documented
□ DNS resolution working (forward + reverse)
□ NTP synchronized (ntpd running)
□ Hardware on VMware HCL
□ Minimum specs met (CPU, RAM, storage)
□ SSH enabled for deployment
□ No stale vSAN metadata on disks

Storage Verification

□ vSAN: Minimum hosts for FTT policy
□ vSAN: Cache and capacity disks identified
□ FC: LUNs presented to all hosts
□ FC: VMFS datastore mounted on all hosts
□ NFS: Share accessible from all hosts

D.2 Workload Domain Deployment Checklist

□ ESXi hosts installed with supported version
□ Hosts commissioned into VCF instance
□ Network pools created for workload domain
□ License entitlements available
□ Storage prerequisites met
□ NSX shared or dedicated (determined)

D.3 Upgrade Pre-Flight Checklist

□ VCF Compatibility Matrix verified
□ Current version documented
□ Target version compatibility confirmed
□ Bundles downloaded and available
□ Management component backups taken
□ VM snapshots created where applicable
□ VCF Health shows all green
□ Maintenance window scheduled
□ Stakeholders notified
□ Rollback plan documented
□ Support contact information ready

Appendix E: Troubleshooting Decision Trees

E.1 VM Connectivity Issues

VM Cannot Communicate
          │
          ▼
    Can VM ping gateway?
          │
    ┌─────┴─────┐
   YES         NO
    │           │
    ▼           ▼
Physical    Check vmk0
Network     Check vSwitch
Issue       Check VLAN
    │           │
    ▼           ▼
Check       Run NSX
ToR         Traceflow
Switch          │
                ▼
          DFW Rule
          Blocking?
              │
        ┌─────┴─────┐
       YES         NO
        │           │
        ▼           ▼
    Check       Check
    Security    TEP/MTU
    Policies    Config

E.2 vSAN Health Issues

vSAN Objects Non-Compliant
          │
          ▼
    Check vSAN Health
    Dashboard
          │
          ▼
    What does it show?
          │
    ┌─────┼─────┐
    │     │     │
    ▼     ▼     ▼
  Host  Disk  Policy
  Down  Full  Issue
    │     │     │
    ▼     ▼     ▼
Replace  Add   Modify
Host    Disks  Policy
    │     │     │
    ▼     ▼     ▼
  Wait for Resync

E.3 SDDC Manager UI Inaccessible

Cannot Access SDDC Manager UI
          │
          ▼
    Ping SDDC Manager IP
          │
    ┌─────┴─────┐
   OK          FAIL
    │           │
    ▼           ▼
SSH to VM   Check:
works?      - VM Power
    │       - Network
    │       - Firewall
    ▼
Check VCF Services:
systemctl status vcf-services
          │
    ┌─────┴─────┐
Running      Failed
    │           │
    ▼           ▼
Check       Check logs:
Nginx       /var/log/vmware/vcf/
    │           │
    ▼           ▼
Restart:    Fix issue
systemctl   then restart
restart     services
nginx

Appendix F: Best Practices

F.1 Deployment Best Practices

1. Always validate compatibility before deployment or upgrade
2. Use vLCM Images (not baselines) for ESXi management
3. Document all IP addresses, VLANs, and credentials before deployment
4. Test DNS forward and reverse resolution for all components
5. Configure NTP consistently across all components
6. Use jumbo frames (MTU 9000) for vSAN and vMotion traffic
7. Ensure MTU 1600+ for NSX overlay TEP traffic

F.2 Operations Best Practices

1. Always change passwords through VCF (not directly on components)
2. Enable automatic password rotation (30, 60, or 90 days)
3. Monitor VCF Health dashboard daily
4. Schedule regular VCF Diagnostics runs
5. Keep VCF Operations updated with latest patches
6. Review and act on VCF Operations recommendations

F.3 Upgrade Best Practices

1. Follow the upgrade order: SDDC Manager → vCenter → NSX → ESXi → vSAN → VCF Ops
2. Take backups/snapshots before each component upgrade
3. Download all bundles before starting upgrade window
4. Verify VCF Health is green before starting
5. Schedule adequate maintenance window
6. Have rollback plan documented and tested

F.4 Security Best Practices

1. Replace default passwords immediately after deployment
2. Use external CA certificates for production environments
3. Enable DFW for micro-segmentation
4. Implement least-privilege access for all users
5. Monitor audit logs through VCF Operations for Logs
6. Keep components patched with latest security updates

Appendix G: Recovery Procedures

G.1 SDDC Manager Recovery

Database Corruption

# 1. Stop VCF services
systemctl stop vcf-services

# 2. Restore PostgreSQL from backup
# (Backup location varies by environment)

# 3. Restart services
systemctl start vcf-services

Service Won't Start

# 1. Check specific service logs
tail -100 /var/log/vmware/vcf/<service>/<service>.log

# 2. Check disk space
df -h

# 3. Check memory
free -m

# 4. Restart individual service
systemctl restart <service-name>

G.2 vCenter Recovery

From VAMI Backup

1. Deploy new vCenter appliance
2. Select "Restore" during deployment
3. Provide backup location and credentials
4. Complete restore wizard

Service Recovery

# Connect via SSH
ssh root@<vcenter-fqdn>

# Check all services
service-control --status --all

# Restart failed service
service-control --restart <service-name>

# Or restart all (causes outage)
service-control --restart --all

G.3 NSX Manager Recovery

Single Node Failure (3-Node Cluster)

1. Cluster continues operating on 2 nodes
2. Deploy replacement NSX Manager
3. Add to existing cluster
4. Wait for cluster sync

Complete Cluster Recovery

1. Restore from NSX backup
2. Reconfigure transport nodes if needed
3. Verify all host connectivity

G.4 ESXi Host Recovery

Disconnected from vCenter

# SSH to host
ssh root@<esxi-host>

# Check vpxa agent
/etc/init.d/vpxa status

# Restart vpxa
/etc/init.d/vpxa restart

# If still disconnected, reconnect from vCenter

Rebuilding Host

1. Install ESXi from ISO
2. Configure management network
3. Commission into VCF
4. Add to workload domain

Appendix H: Additional Exam Questions

VCF Automation Questions (Continued)

Q: An administrator creates a new Organization for All Applications. What NSX constructs are automatically configured during regional networking creation? - A) Outbound SNAT rule - B) Default VPC - C) Provider Tier-0 Gateway - D) VPC connectivity profile - E) All of the above

Answer: E - All four constructs are automatically created.

Q: Which VCF Automation component provides self-service catalog aggregation? A) Assembler B) Service Broker C) Orchestrator D) Code Stream

Answer: B - Service Broker aggregates blueprints, catalog items, and templates.

Q: What deployment model limitation exists for VCF Automation Simple deployments? A) Cannot have more than 100 VMs B) Cannot scale up in place - must redeploy C) Cannot integrate with NSX D) Cannot use external databases

Answer: B - Simple deployments cannot be upgraded to HA in place.

VCF Operations Questions (Continued)

Q: What is the default cycle time in VCF Operations for wait and cancel cycles? A) 5 minutes B) 10 minutes C) 20 minutes D) 30 minutes

Answer: C - Default cycle time is 20 minutes.

Q: Which VCF Operations feature enables Crown Jewel Analysis for identifying critical applications? A) VCF Operations for Logs B) VCF Operations for Networks C) VCF Operations Collector D) VCF Operations Fleet Management

Answer: B - VCF Operations for Networks provides Crown Jewel Analysis.

Q: What vCenter version is required for configuration drift detection? A) vCenter 7.0 U3+ B) vCenter 8.0 U2+ C) vCenter 8.0 U3+ D) vCenter 9.0+

Answer: C - vCenter 8.0 U3 or later is required.

NSX Questions (Continued)

Q: What is the minimum MTU required for NSX overlay networks? A) 1500 bytes B) 1550 bytes C) 1600 bytes D) 9000 bytes

Answer: C - 1600 bytes minimum (GENEVE adds ~54 bytes overhead).

Q: Which NSX credentials are managed by VCF? A) admin, root, backup B) admin, audit, backup C) admin, audit, root D) root, audit, backup

Answer: B - admin, audit, and backup accounts are VCF-managed.

Q: What is required before creating a VLAN-backed segment in NSX? A) Tier-1 Gateway B) VLAN ID and VLAN Transport Zone C) Overlay Transport Zone D) DHCP server

Answer: B - VLAN ID and VLAN Transport Zone are required.

vSAN Questions (Continued)

Q: What is the minimum number of hosts for RAID-6 with auto-policy in vSAN ESA? A) 4 hosts B) 6 hosts C) 8 hosts D) 10 hosts

Answer: C - 8 hosts minimum for RAID-6 with auto-policy.

Q: Which vSAN architecture is recommended for nested/lab environments? A) vSAN ESA B) vSAN OSA C) vSAN HCI Mesh D) vSAN Stretched Cluster

Answer: B - vSAN OSA works with virtualized storage; ESA requires physical NVMe.

Q: What VCF feature provides NFS/SMB access to both VMs and external clients? A) vSAN Storage Cluster B) vSAN File Services C) vSAN Stretched Cluster D) Content Library

Answer: B - vSAN File Services provides file shares.

Lifecycle Questions (Continued)

Q: Which component must be upgraded first in VCF? A) vCenter B) NSX C) SDDC Manager D) ESXi

Answer: C - SDDC Manager must always be upgraded first.

Q: What tool is used to download bundles for air-gapped VCF deployments? A) SDDC Manager B) VCF Installer C) VCF Download Tool D) Broadcom Portal API

Answer: C - VCF Download Tool retrieves bundles for offline installation.

Q: What replaces Cloud Builder in VCF 9.0? A) SDDC Manager B) VCF Installer C) VCF Operations D) VCF Automation

Answer: B - VCF Installer replaces Cloud Builder.

Appendix I: Glossary

Document Total Pages: ~200 Total Procedures: 150+ Total Study Questions: 200+

End of VMware Cloud Foundation 9.0 Professional Handbook