NVIDIA Mission Control Software with GB200/GB300 NVL72 Systems Administration Guide

NVIDIA Mission Control

NVIDIA Mission Control

• Overview
  • System Design
  • Management Servers
  • Control Plane Services Distribution
• Mission Control Software Stack
  • Cluster Management
  • User Management
  • Integrating External LDAP
• Node and Category Management
  • Introduction to Categories
  • Overview of BaseOS for NMC DGX Systems
  • Category for DGX Compute Nodes Using BaseOS
  • Slurm Roles and Configuration Overlays
  • Integrating categories with Slurm Roles and Configuration Overlays
• Slurm Workload Management
  • Slurm and IMEX
  • Slurm Topology Block Support
  • Slurm Partitioning with NVLink
  • Common Slurm Management Tasks
  • Example: Running Interactive Slurm Job to confirm IMEX setup
• NVIDIA Run:ai Installation
  • NVIDIA Run:ai Deployment
  • Node Categories
  • NVIDIA Run:ai Validation
• Adding and Removing Nodes from Run:ai or Slurm
  • Overview
  • Prerequisites
  • Adding a compute node to an existing Run:ai cluster
  • Adding a compute node to an existing Slurm cluster
• Observability Software
  • Base Command Manager Monitoring
  • Prometheus and Grafana
  • Logs
• Connecting to NVIDIA Mission Control Autonomous Hardware Recovery
  • Dashboard
• Cluster Validation
  • Entrypoint of Automated Testing
  • Firmware checks
  • Reports of Testings
• Health Checks & Alerts
  • Alarms Dashboard
  • BCM Slurm Job Lifecycle Checks (Prolog and Epilog)
  • Periodic Health Checks (Alarms)
  • NVIDIA Mission Control autonomous hardware recovery Alarm Configuration
  • Alarm States
• Firmware Upgrades with NVIDIA Mission Control autonomous hardware recovery
• NVIDIA Mission Control autonomous hardware recovery Break/Fix Workflow
  • Key Features
  • Entrypoint of Break/Fix Workflow
  • Break/Fix Workflow Components
• NVIDIA Mission Control autonomous hardware recovery Break/Fix Post RMA
  • Key Features
  • Post RMA Workflow Components
  • Post RMA Workflow Results
• NVIDIA Mission Control autonomous hardware recovery Domain Triage
  • Domain Triage Workflow Components
• NVIDIA Mission Control autonomous hardware recovery Break/Fix Switch Post RMA
  • Switch Post RMA Introduction
  • Switch Post RMA Workflow Components
• Out-of-Band Management
  • Using cmsh to get BMC events
  • Using cmsh to access BMC SoL
  • Using cmsh to get BMC IPs
• High-Speed Fabric Management
  • NVLink and NVLink Switch
  • NVLink Management Software (NMX)
  • InfiniBand Fabrics
• Leak Detection
  • Leak Example in cmsh
  • Leak Example in BCM UI
• Backups
  • Cluster Installation Backup
  • Local Database and Data Backups and Restoration

Power Reservation Steering

Power Reservation Steering

• Introduction
  • Motivation
  • What is PRS?
• Concepts and Components
  • Concepts
  • Components
• Installation
• Advanced Configuration
  • Understanding Roles and Overlays
  • Configuring PRS Server
  • Configuring PRS Client
• Metrics
  • Device-Level Metrics
  • Node-Level Metrics
  • PD-Level Metrics
  • Job-Level Metrics
  • Metrics with cmsh
  • Metrics with BaseView (Web UI)
• Troubleshooting
  • Jobs stuck in the queue pending
• FAQ
  • Static vs. dynamic power
  • Who should configure the PDN?
  • Why can’t I submit one large job across all nodes?
  • Example 1: Configuring a PD for a GB200 NVL72 rack
  • Example 2: Power budget adjustment and verification for two GB200 NVL72 racks
  • Will PRS PDs update when a node is removed from BCM or a category?
  • Will PRS PDs update when a new node is added to a category?
  • What is the lifecycle of updates to the PRS config server?
  • Does PRS reflect autoscaler changes automatically?
  • When is the CPU included in PRS-managed devices?
  • What does it mean to stop or start a PD?

Autonomous Recovery Engine

Autonomous Recovery Engine

• Introduction
• Installation Process
• Configuration
• Accessing Clusters
• Accessing Dashboards
• Monitoring and Logs
• Grafana Cloud Setup
• Installing and Upgrading ARE
• Example Commands
• Viewing Job Details
• Accessing the Cockpit
  • Monitoring and Performance
• ARE Job Monitoring
  • Application Log Format Requirements
• Confirming ARE is Operational
  • Singleton Dependency
  • Requeue
• How-to: Toggle Dry-Run Mode
• Debugging Common Issues
  • Issue: Job is not managed by ARE or is not visible on the Cockpit
  • Issue: Job is available in Cockpit but does not receive any anomaly and checkpoint events
  • Issue: The job died but was not requeued
  • Issue: The requeued job or the next dependency job was not released
• Administrative and Operational Guide for AJR NMC 2.0 Release
  • Introduction
  • Installation Process
  • Configuration
  • Accessing Clusters
  • Accessing Dashboards
  • Monitoring and Logs
  • Grafana Cloud Setup
  • Viewing Job Details from Job Manager
  • Accessing the Cockpit
  • AJR Monitoring the Job
  • Testing AJR Functionality
  • How to turn on and off the dry-run mode
  • Debugging Some Common Issues
  • Deployment Requests
  • Conclusion

Workload Power Profile Solution (WPPS)

• Introduction
  • Motivation
• Components and Concepts
  • Core Components
  • Scalability and Performance
  • Concepts
• Installation
• First Slurm Job with WPPS
  • Case 1
  • Case 2
• Frequently Asked Questions