NVIDIA Mission Control Management Plane and Rack Setup with NVIDIA GB200 NVL72 Systems Installation Guide

Installation Guide

• Introduction
• BCM Software Installation
  • Acquire BCM ISO
  • BCM Software Installation
    • Method 1—Remote Installation
    • Method 2—Local Installation
    • Server Boot Settings
    • Install the Software
• BCM Networking Setup
  • Manual BCM Networking Setup
    • internalnet
    • dgxnet
    • ipminet
    • computenet
    • storagenet
    • failovernet
    • globalnet
  • bcm-netautogen
• Configuration for Node Provisioning
  • Mixed Architecture, Software Image, and Category Setup
  • Control Plane Node Configuration
  • GB200/GB300 Rack Node Configuration
  • Final Setup and Provisioning
• High Availability
  • CMHA Setup
  • NFS Configuration
  • Shared Storage Setup
  • Verify HA Failover Functionality
  • Post HA and NFS Configuration Steps
    • Mixed Architecture Fsmounts Configuration
    • Increase Retry Count to Ensure /home and /cm/shared Get Mounted
• GB200/GB300 Rack Power On and Bring Up
  • Initial Power On and Provisioning
    • GB200/GB300 Compute Tray Power On and Provisioning
    • NVLink Switch Tray Power On and Configuration
    • NVLink Switch ZTP Process Execution
• GB200/GB300 Rack Firmware Update
  • Rack Firmware Updates
    • Compute Tray Firmware Update Process
    • Power Cycle Methods
    • NVLink Switch Firmware Update Process
    • NVLink Switch: Updating NVOS
  • Power Shelf Firmware Update Process
  • BlueField and CX7 FW Update Process
  • Combined CX-7 and BlueField Update
  • DGX OS Update
  • Operational Security Requirements
• Deployment Summary Validation Checklist
  • Prerequisites and Hardware Validation
    • Physical Infrastructure
  • BCM Software Installation Validation
    • Head Node Configuration
  • Network Configuration Validation
    • Network Definitions
  • Software Image and Category Validation
    • Mixed Architecture Setup
  • Control Plane Node Configuration Validation
    • Head Node Configuration
    • SLURM Login Nodes
    • K8s-system-admin Nodes
    • K8s-system-user Nodes
  • UFM Pre-Setup Validation
  • GB200/GB300 Rack Configuration Validation
    • Rack Import Verification
    • GB200/GB300 Compute Tray Configuration
    • NVLink Switch Configuration
  • High Availability Configuration Validation
    • HA Setup Verification
    • NFS and Shared Storage
  • Rack Power-On and Provisioning Validation
    • GB200/GB300 Compute Tray Bring-Up
    • NVLink Switch Bring-Up
    • Power Shelf Validation
  • Firmware Update Validation
    • Compute Tray Firmware
    • Firmware Management Setup
    • NVLink Switch Firmware
    • BlueField and CX7/CX8 Firmware
    • Power Shelf Firmware
  • System Integration and Final Validation
    • Network Connectivity
    • Compute Workload Validation
    • Monitoring and Management
    • Security and Access Control
  • Documentation and Handover
    • System Documentation
  • Performance Validation
    • System Performance Benchmarks
  • Final System Acceptance
    • Deployment Sign-Off
  • Commands for Quick Final Validation
    • System Status Overview
    • Category and Software Image Verification
    • Network Configuration Validation
    • Node Interface Validation
    • Network Connectivity Validation
    • Power and BMC Validation
    • Firmware Validation
    • Health and Monitoring Validation
    • High Availability Validation
    • Provisioning Validation
  • Critical Success Criteria
    • Infrastructure Criteria
    • Software Criteria
    • Integration Criteria
    • Operational Criteria
• Appendix
  • Understanding Netnames
  • Hardware Information Gathering
    • Networking (NIC) Information
    • Finding Out More About Local NVIDIA Networking Cards
    • How to Find Physical Port Mapping to netname
    • Disk Information
  • Other Tips and Tricks
    • Check the DGX OS packages to meet current SBOM
    • Creating a RAID 1 Software Partition
    • Switching the Network Mode for CX-7/BF3 Cards
    • Accessing GB200 Compute Tray using Minicom (USB to Micro-USB Connection)
    • Set Jump host using Firefox and Foxy Proxy
    • Jailbreak a UFM Appliance
    • Updating the GB200/GB300 compute tray Bluefield 3 Option ROM
  • KVM Setup on Ubuntu
    • Prerequisites
    • Network Bridge Setup
    • Guest KVM Creation
    • Console Access
    • Clean Up Guest KVM