NVIDIA Mission Control Software Systems Administration Guide

NVIDIA Mission Control

NVIDIA Mission Control

• Overview
  • Target Audience
  • Prerequisites
  • System Design
  • Mission Control Architecture Overview
  • Key Components by Architecture
  • Infrastructure Services
  • Advanced Features
  • Network and Storage Infrastructure
  • Administrative Tasks
  • Architecture Selection Guide
  • Document Conventions
• Mission Control Software Stack
  • Cluster Management
  • User Management
  • Integrating External LDAP
• Airgap Operations
  • Architecture
  • Examples
• Node and Category Management
  • Introduction to Categories
  • Overview of BaseOS for NMC DGX Systems
  • Category for DGX Compute Nodes Using BaseOS
  • Slurm Roles and Configuration Overlays
  • Integrating categories with Slurm Roles and Configuration Overlays
• Slurm Workload Management
  • Slurm and IMEX
  • Slurm Topology Block Support
  • Slurm Partitioning with NVLink
  • Common Slurm Management Tasks
  • Example: Running Interactive Slurm Job to confirm IMEX setup
• NVIDIA Run:ai Installation
  • NVIDIA Run:ai Deployment
  • Node Categories
  • NVIDIA Run:ai Validation
• Adding and Removing Nodes from Run:ai or Slurm
  • Overview
  • Prerequisites
  • Adding or removing a compute node from a Run:ai cluster
  • Adding or removing a compute node from a Slurm cluster
• Observability Software
  • Base Command Manager Monitoring
  • Prometheus and Grafana
  • Logs
• Connecting to NVIDIA Mission Control autonomous hardware recovery
  • Baseline testing
  • Firmware upgrades
  • Break/Fix
  • Post-RMA
  • Domain Triage
  • Health checks and operations
  • Platform support
  • Automated Health Checks with NVIDIA Mission Control autonomous hardware recovery
  • NVIDIA Mission Control autonomous hardware recovery Active Node Updates
  • NVIDIA Mission Control autonomous hardware recovery UI Tips and Troubleshooting
• Out-of-Band Management
  • Using cmsh to get BMC events
  • Using cmsh to access BMC SoL
  • Using cmsh to get BMC IPs
• NVLink Partition Management
  • Adapting NVLink Partitions
  • Maintenance Flows for Compute Trays
  • Workload Management and Job Scheduling Considerations
• NVLink Management Software (NMX + NetQ)
• Leak Detection
  • Leak Example in cmsh
  • Leak Example in BCM UI
  • Leak Related Metrics
• Backups
  • Cluster Installation Backup
  • Local Database and Data Backups and Restoration
• Autonomous Job Recovery
  • Introduction
  • Accessing Clusters
  • Accessing Dashboards
  • Monitoring and Logs
  • Grafana Cloud Setup
  • Viewing Job Details
  • Accessing the Cockpit
  • AJR Job Monitoring
  • Confirming AJR is Operational
  • How-to: Toggle Dry-Run Mode
  • Debugging Common Issues
• Power Reservation Steering
  • Introduction
  • Concepts and Components
  • Installation
  • Advanced Configuration
  • Metrics
  • Troubleshooting
  • FAQ