NVIDIA DGX SuperPOD: Administration Guide Featuring NVIDIA DGX H100 and DGX A100 Systems

Document Number	DU-10263-001 V05
Publication Date	2023-06-04
Base Command Manager Version	3.23.05

Tip

The NVIDIA DGX SuperPOD: Administration Guide Featuring NVIDIA DGX H100 and DGX A100 Systems is also available as a PDF.

Administration Guide

• Overview
  • System Design
  • Management Servers
• Cluster Management
  • Concepts
    • Devices
    • Software Images
    • Node Categories
    • Node Groups
    • Roles
  • Authentication
    • Changing Administrative Passwords
    • ssh Logins
    • Certificates
      • PEM Certificates and CMDaemon Front-end Authentication
      • Using cmsh and Authenticating to CMDaemon
      • Using Base View and Authenticating to the Cluster Manager
    • Profiles
  • Base View GUI
    • Cluster Management GUI Service
      • Base View Login Window
      • Base View Default Display on Connection
    • Navigating the Cluster with Base View
      • Settings
      • Other Tab Information
  • Cluster Management Shell
    • Invoking cmsh
      • Batch Mode and Piping in cmsh
      • Dotfiles and /etc/cmshrc File for cmsh
      • Defining Command Aliases in cmsh
      • Built-in Aliases in cmsh
      • Automatic Aliases in cmsh
      • Default Arguments in cmsh Scripts
      • cmsh Options
    • Levels, Modes, Help, and Commands Syntax in cmsh
      • Navigation Through Modes and Objects in cmsh
    • Working with Objects
      • use and exit
      • add, commit, and remove
      • clone, modified, and swap
      • get, set, and refresh
      • clear
      • list, format, and sort
      • append and removefrom
      • usedby
      • validate
      • show
      • assign and unassign
      • import
    • Advanced cmsh Features
      • CLI Editing
      • history and timestamps
      • Mixing cmsh and Unix Shell Commands
      • Output Redirection
      • Input Redirection
      • ssh
      • time
      • watch
      • foreach
      • Node List Syntax
      • groupingsyntax
      • range
      • bookmark
      • rename
      • Using CMDaemon Environment Variables in Scripts
      • Converting Tables to JSON with cmsh
• Cluster Management Daemon
  • Controlling CMDaemon
  • Configuring CMDaemon
    • CMDaemon Versions
  • Configuring CMDaemon Logging
    • CMDaemon Logging Configuration Global Debug Mode
    • CMDaemon Subsystem Logging Configuration Debug Mode
      • CMDaemon Subsystem Logging Configuration Severity Levels
      • CMDaemon Subsystem Logging Configuration Deployment
  • Configuration File Modification and the FrozenFile Directive
  • Configuration File Precedence
• User Management
  • Managing Users and Groups with Base View
  • Managing Users and Groups with cmsh
    • Adding a User
    • Saving the Modified State
    • Editing Properties of Users and Groups
      • Editing Users with set and clear
      • Editing Groups with append and removefrom
    • Reverting to the Unmodified State
    • Removing a User
  • LDAP
  • Tokens and Profiles
    • Modifying Profiles
    • Creation of Custom Certificates with Profiles
      • Listing Certificates with cmsh
      • Listing Certificates with Base View
      • Node Certificates
      • Creating a Custom Certificate
      • Creating a New Certificate for cmsh Users
      • Associating Users with Paths to a New Custom Certificate
      • Creating a Custom Certificate for Base View Users
    • Logging the Actions Of CMDaemon Users
      • Creation of Certificates for Nodes with cm component-certificate
    • Compute Node LDAP PEM and Key creation
• Managing Slurm
  • Introduction
  • Checking Node Status
  • Showing Detailed Node Information
  • Draining a Node
  • Updating Slurm Configuration
  • Slurm Prolog and Epilog
    • Details of Prolog and Epilog Scripts
    • Workload Manager Configuration For Prolog and Epilog Scripts
  • Listing Slurm Jobs in the Queue
  • Canceling a Slurm Job
  • Managing the Parameters on a Job
    • Additional Resources
• Monitoring Cluster Devices
  • Basic Monitoring Example and Action
    • Synopsis Of Basic Monitoring Example
    • Setting Up the Pieces
      • Running Artificial Loads
      • Setting Up the Kill Action
    • Using the Basic Monitoring Example
      • Adding the Action to the Actions List
      • Setting Up a Trigger Using CPUUser on the Login Node
      • The Result
  • Monitoring Concepts and Definitions
    • Measurables
      • Types of Measurables
      • no data and Measurables
      • Entities and Measurables
      • Listing Measurables Used by an Entity
      • Listing Measurables from monitoring Mode
      • Viewing Parameters for a measurable in monitoring Mode
      • Enummetrics
      • Metrics
    • Health Check
      • Health Checks
      • All Configured Health Checks
      • Configured Health Checks for an Entity
      • Severity Levels for Health Checks, and Overriding Them
      • Default Templates for Health Checks and Triggers
    • Trigger
    • Action
    • Severity
    • AlertLevel
    • Flapping
    • Data Producer
    • Main Monitoring Interfaces of Base View
  • Monitoring Visualization with Base View
    • The Monitoring Window
      • Finding and Selecting the Measurable to be Plotted
      • Plotting The Measurable
• Managing High-Speed Fabrics
  • Verifying that UFM is Running
• System Health Checks and Debugging
  • Collecting Log Files
    • Log Subsystems
    • Increasing Log Verbosity
    • Global Debug Mode
    • LOGPREFIX
• Provisioning Nodes
  • Role Setup with cmsh
  • Role Setup with Base View
  • Housekeeping
    • Provisioning Node Selection
    • Limiting Provisioning Tasks
    • Provisioning Tasks Deferral and Failure
    • Role Change Notification
    • Role Draining and Undraining Nodes
    • Provisioning Node Update Safeguards
      • Synchronization of fspart Subdirectories to Provisioning Nodes
• Product Security
• Backups
  • Cluster Installation Backup
  • Local Database and Data Backups and Restoration
    • Database Corruption and Repairs
    • Restoring from Local Backup
    • Cloning Databases
    • Cloning Extra Databases

Notices

• Notices
  • Notice
  • Trademarks
  • Copyright