NVIDIA Mission Control Software Systems Administration Guide

NVIDIA Mission Control

NVIDIA Mission Control

• Overview
  • Target Audience
  • Prerequisites
  • System Design
  • Mission Control Architecture Overview
  • Key Components by Architecture
  • Infrastructure Services
  • Advanced Features
  • Network and Storage Infrastructure
  • Administrative Tasks
  • Architecture Selection Guide
  • Document Conventions
• Mission Control Software Stack
  • Cluster Management
  • User Management
  • Integrating External LDAP
• Node and Category Management
  • Introduction to Categories
  • Overview of BaseOS for NMC DGX Systems
  • Category for DGX Compute Nodes Using BaseOS
  • Slurm Roles and Configuration Overlays
  • Integrating categories with Slurm Roles and Configuration Overlays
• Slurm Workload Management
  • Slurm and IMEX
  • Slurm Topology Block Support
  • Slurm Partitioning with NVLink
  • Common Slurm Management Tasks
  • Example: Running Interactive Slurm Job to confirm IMEX setup
• NVIDIA Run:ai Installation
  • NVIDIA Run:ai Deployment
  • Node Categories
  • NVIDIA Run:ai Validation
• Adding and Removing Nodes from Run:ai or Slurm
  • Overview
  • Prerequisites
  • Adding or removing a compute node from a Run:ai cluster
  • Adding or removing a compute node from a Slurm cluster
• Observability Software
  • Base Command Manager Monitoring
  • Prometheus and Grafana
  • Logs
• Connecting to NVIDIA Mission Control Autonomous Hardware Recovery
• GB300 (Limited Feature Support)
  • Firmware Upgrades with NVIDIA Mission Control autonomous hardware recovery
• GB200
  • Dashboard
  • Cluster Validation
  • Health Checks & Alerts
  • Firmware Upgrades with NVIDIA Mission Control autonomous hardware recovery
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Workflow
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Post RMA
  • NVIDIA Mission Control autonomous hardware recovery Domain Triage
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Switch Post RMA
• B200
  • Dashboard
  • Cluster Validation
  • Automated Health Checks with NVIDIA Mission Control autonomous hardware recovery
  • Firmware Upgrades with NVIDIA Mission Control autonomous hardware recovery
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Workflow
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Post RMA
  • NVIDIA Mission Control autonomous hardware recovery Domain Triage
  • NVIDIA Mission Control autonomous hardware recovery Break/Fix Switch Post RMA
• Out-of-Band Management
  • Using cmsh to get BMC events
  • Using cmsh to access BMC SoL
  • Using cmsh to get BMC IPs
• NVLink Partition Management
  • Adapting NVLink Partitions
  • Maintenance Flows for Compute Trays
  • Workload Management and Job Scheduling Considerations
• NVLink Management Software (NMX + NetQ)
• Leak Detection
  • Leak Example in cmsh
  • Leak Example in BCM UI
  • Leak Related Metrics
• Backups
  • Cluster Installation Backup
  • Local Database and Data Backups and Restoration
• Autonomous Job Recovery
  • Introduction
  • Installation Process
  • Configuration
  • Accessing Clusters
  • Accessing Dashboards
  • Monitoring and Logs
  • Grafana Cloud Setup
  • Installing and Upgrading AJR
  • Example Commands
  • Viewing Job Details
  • Accessing the Cockpit
  • AJR Job Monitoring
  • Confirming AJR is Operational
  • How-to: Toggle Dry-Run Mode
  • Debugging Common Issues
• Power Reservation Steering
  • Introduction
  • Concepts and Components
  • Installation
  • Advanced Configuration
  • Metrics
  • Troubleshooting
  • FAQ
• Workload Power Profile Solution (WPPS)
  • Introduction
  • Components and Concepts
  • Installation
  • First Slurm Job with WPPS
  • Frequently Asked Questions