NVIDIA DGX BasePOD: Deployment Guide Featuring NVIDIA DGX H200/H100 Systems

Document Number	NA
Publication Date	2025-2-22

Tip

The NVIDIA DGX BasePOD: Deployment Guide Featuring NVIDIA DGX H200 and H100 Systems.

Overview

• Introduction
• Hardware Overview
• Networking
  • DGX H200/H100 System Network Ports
  • DGX BasePOD Network Overview
  • Managementnet and External networks
  • oobmanagementnet (ipminet)
  • externalnet uplink
  • computenet (ibnet)
• Software Overview
  • Kubernetes (K8s)
• NFS Storage

BCM/Network Deployment

• BCM Introduction
• Network Deployment
  • SN4600C – managementnet ethernet switches
    • SN4600C-1 reference configuration
    • SN4600C-2 reference configuration
  • SN2201 – IPMI Switch for Out-of-Band Management
  • Computenet Configuration
  • QM9700 IB Switches
    • QM-9700-1
    • QM-9700-2
• BCM Headnodes Pre-install Preparation
  • NFS server
  • Verify DNS & NTP servers
  • Cluster Nodes Configuration
    • DGX BIOS Config and Network Interface Boot Order
    • Control Plane and Workload Management Nodes
    • Other Branded appliances
    • RAID/Storage Configuration
• BCM Headnodes Installation
  • Download the Base Command Manager ISO
  • Primary Headnode preparation
  • Dell appliances with iDRAC9
  • Other Vendor Appliances
  • Booting the Base Command Manager Graphical Installer
  • Login to the Headnode
  • Update BCM
  • Activate the BCM Cluster License
  • Enable Bonding on the Headnode
• Cluster Bring Up
  • Enable DeviceResolveAnyMAC
  • Define Cluster Networks
    • Change network names
    • Computenet Config
    • Storagenet Config
    • IB Storage
  • Enable Out-of-band Management of Cluster Nodes
  • DGX Node Bringup
    • Software Image Setup for DGX’s
    • DGX Node category setup
    • DGX Interface Definitions
    • Defining the storage fabric:
    • Ethernet (tcp)
    • Infiniband (o2ib)
    • Define DGX-01’s MAC address
    • Test Provisioning of DGX-01

BCM HA and NFS

• BCM HA
• Setup shared NFS storage.
• Testing BCM HA

Deploying Workload Manager

• Deploy Kubernetes
  • Kubernetes Node Setup
  • Kubernetes Deployment
  • Add a kubernetes user
• Compute network/ IB Interfaces Configuration
  • Validate IB/Compute interfaces
  • Configure SR-IOV NetworkNodePolicy CR

Cluster Validation

• Validate the GPU status/health
• Validate the system topology/NVlink
• Validate the GPU/RDMA access within the container
  • Validate GPU access from container
  • Validate the RDMA Network from the container
• Validate the node level NCCL test with 8 GPUs
• Validate the cluster level NCCL test with 4 nodes and 32 GPUs

Appendix

• Site Survey
  • Sample Site Survey
  • Blank Site Survey
• Switch Configurations
  • SN2201 (oobmanagementnet) Switch Configuration
  • SN4600C-1 (managementnet) Switches Configuration
    • SN4600C-1 Configuration
    • SN4600C-2 Configuration
  • QM9700 (computenet) Switches Configuration
    • QM9700-1
    • QM9700-2