NVIDIA Mission Control Software Systems Administration Guide

NVIDIA Mission Control

NVIDIA Mission Control

• Overview
  • Target Audience
  • Prerequisites
  • System Design
  • Mission Control Architecture Overview
  • Key Components by Architecture
  • Infrastructure Services
  • Advanced Features
  • Network and Storage Infrastructure
  • Administrative Tasks
  • Architecture Selection Guide
  • Document Conventions
• Mission Control Software Stack
  • Cluster Management
  • User Management
  • Integrating External LDAP
• Node and Category Management
  • Introduction to Categories
  • Overview of BaseOS for NMC DGX Systems
  • Category for DGX Compute Nodes Using BaseOS
  • Slurm Roles and Configuration Overlays
  • Integrating categories with Slurm Roles and Configuration Overlays
• Slurm Workload Management
  • Slurm and IMEX
  • Slurm Topology Block Support
  • Slurm Partitioning with NVLink
  • Common Slurm Management Tasks
  • Example: Running Interactive Slurm Job to confirm IMEX setup
• NVIDIA Run:ai Installation
  • NVIDIA Run:ai Deployment
  • Node Categories
  • NVIDIA Run:ai Validation
• Adding and Removing Nodes from Run:ai or Slurm
  • Overview
  • Prerequisites
  • Adding or removing a compute node from a Run:ai cluster
  • Adding or removing a compute node from a Slurm cluster
• Observability Software
  • Base Command Manager Monitoring
  • Prometheus and Grafana
  • Logs
• Connecting to NVIDIA Mission Control Autonomous Hardware Recovery
• GB300 (Limited Feature Support)
  • Firmware Upgrades with NVIDIA Mission Control autonomous hardware recovery
• GB200
  • Dashboard
  • Cluster Validation
  • Health Checks & Alerts
  • Firmware Upgrades with NVIDIA Mission Control autonomous hardware recovery
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Workflow
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Post RMA
  • NVIDIA Mission Control autonomous hardware recovery Domain Triage
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Switch Post RMA
• B200
  • Dashboard
  • Cluster Validation
  • Automated Health Checks with NVIDIA Mission Control autonomous hardware recovery
  • Firmware Upgrades with NVIDIA Mission Control autonomous hardware recovery
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Workflow
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Post RMA
  • NVIDIA Mission Control autonomous hardware recovery Domain Triage
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Switch Post RMA
• Out-of-Band Management
  • Using cmsh to get BMC events
  • Using cmsh to access BMC SoL
  • Using cmsh to get BMC IPs
• NVLink Partition Management
  • Adapting NVLink Partitions
  • Maintenance Flows for Compute Trays
  • Workload Management and Job Scheduling Considerations
• NVLink Management Software (NMX + NetQ)
• Leak Detection
  • Leak Example in cmsh
  • Leak Example in BCM UI
  • Leak Related Metrics
• Backups
  • Cluster Installation Backup
  • Local Database and Data Backups and Restoration

Power Reservation Steering

Power Reservation Steering

• Introduction
  • Motivation
  • What is PRS?
• Concepts and Components
  • Concepts
  • Components
• Installation
• Advanced Configuration
  • Understanding Roles and Overlays
  • Configuring PRS Server
  • Configuring PRS Client
• Metrics
  • Device-Level Metrics
  • Node-Level Metrics
  • PD-Level Metrics
  • Job-Level Metrics
  • Metrics with cmsh
  • Metrics with BaseView (Web UI)
• Troubleshooting
  • Jobs stuck in the queue pending
• FAQ
  • Static vs. dynamic power
  • Who should configure the PDN?
  • Why can’t I submit one large job across all nodes?
  • Example 1: Configuring a PD for a GB200/GB300 NVL72 rack
  • Example 2: Power budget adjustment and verification for two GB200 NVL72 racks
  • Will PRS PDs update when a node is removed from BCM or a category?
  • Will PRS PDs update when a new node is added to a category?
  • What is the lifecycle of updates to the PRS config server?
  • Does PRS reflect autoscaler changes automatically?
  • When is the CPU included in PRS-managed devices?
  • What does it mean to stop or start a PD?

Autonomous Job Recovery

Autonomous Job Recovery

• Introduction
• Installation Process
• Configuration
• Accessing Clusters
• Accessing Dashboards
• Monitoring and Logs
• Grafana Cloud Setup
• Installing and Upgrading AJR
• Example Commands
• Viewing Job Details
• Accessing the Cockpit
  • Monitoring and Performance
• AJR Job Monitoring
  • Application Log Format Requirements
• Confirming AJR is Operational
  • Singleton Dependency
  • Requeue
• How-to: Toggle Dry-Run Mode
• Debugging Common Issues
  • Issue: Job is not managed by AJR or is not visible on the Cockpit
  • Issue: Job is available in Cockpit but does not receive any anomaly and checkpoint events
  • Issue: The job died but was not requeued
  • Issue: The requeued job or the next dependency job was not released

Workload Power Profile Solution (WPPS)

• Introduction
  • Motivation
• Components and Concepts
  • Core Components
  • Scalability and Performance
  • Concepts
• Installation
• First Slurm Job with WPPS
  • Case 1
  • Case 2
• Frequently Asked Questions