NVIDIA DGX Cloud Run:ai Documentation

Run:ai on DGX Cloud Manual

• 1. Product Overview
  • 1.1. Cluster Architecture
  • 1.2. Shared Responsibility Model
    • 1.2.1. NVIDIA Responsibilities
    • 1.2.2. End User Responsibilities
  • 1.3. Cluster Ingress and Egress
  • 1.4. Cluster User Scopes and User Roles
    • 1.4.1. User Scopes
      • 1.4.1.1. Run:ai Departments
      • 1.4.1.2. Run:ai Projects
    • 1.4.2. Cluster Users
      • 1.4.2.1. Customer Admin Roles and NVIDIA Admin Roles
        • 1.4.2.1.1. Customer Admin Roles
        • 1.4.2.1.2. NVIDIA Admin Roles
      • 1.4.2.2. Customer User Roles
  • 1.5. Next Steps
• 2. Cluster Onboarding Guide
  • 2.1. Introduction
  • 2.2. Admin Steps
    • 2.2.1. Overview of Personas in DGX Cloud
    • 2.2.2. Accessing the DGX Cloud Run:ai Cluster
    • 2.2.3. Setting up the Run:ai and Kubernetes CLI
    • 2.2.4. Creating a Department in Run:ai
    • 2.2.5. Creating a Project in Run:ai
    • 2.2.6. Assigning User Roles in Run:ai
    • 2.2.7. Creating an Environment
    • 2.2.8. Creating a Data Source
  • 2.3. User Steps
    • 2.3.1. Running Sample Workloads
      • 2.3.1.1. Running a Single Node NCCL Test
        • 2.3.1.1.1. Prerequisites
        • 2.3.1.1.2. Instructions
      • 2.3.1.2. Running a Multi-Node NCCL Test
        • 2.3.1.2.1. Prerequisites
        • 2.3.1.2.2. Instructions
      • 2.3.1.3. (Optional) Running a Single Node NeMo Interactive Workload
      • 2.3.1.4. (Optional) Downloading Data into the Data Source
  • 2.4. Conclusion
• 3. Cluster Administrator Guide
  • 3.1. Cluster Handover
  • 3.2. Accessing your Cluster
    • 3.2.1. Accessing the Run:ai UI
  • 3.3. Managing Users
    • 3.3.1. Creating Users
      • 3.3.1.1. Assigning Roles in Run:ai
      • 3.3.1.2. Access Rule Creation in Run:ai
        • 3.3.1.2.1. Example: Assigning Access Roles
  • 3.4. Managing Departments and Projects
    • 3.4.1. Departments
      • 3.4.1.1. Modifying the Default Department
      • 3.4.1.2. Creating a Department
    • 3.4.2. Projects
      • 3.4.2.1. Creating a Project
      • 3.4.2.2. Editing a Project
      • 3.4.2.3. Updating Access to Projects
  • 3.5. Managing Compute Resources
    • 3.5.1. Creating a New Compute Resource
  • 3.6. Managing Your Storage Utilization (CLI)
    • 3.6.1. Volume Protection
  • 3.7. Retrieving the Kubernetes Configuration File (CLI)
  • 3.8. Configuring the Ingress/Egress CIDRs for the Cluster (CLI)
  • 3.9. Exporting Kubernetes API Server Audit Logs to External Storage (CLI)
    • 3.9.1. Get Writer Principal from CRD
    • 3.9.2. Create Object Bucket in Customer Account
    • 3.9.3. Update CRD to Set Destination Field
    • 3.9.4. Check Audit Logs in Cloud Bucket
  • 3.10. Restricting Container Image Access
  • 3.11. Managing Cluster Updates and Upgrades
• 4. Cluster User Guide
  • 4.1. Logging Into Your Cluster
    • 4.1.1. Accessing the Run:ai UI
    • 4.1.2. Overview
  • 4.2. Understanding Your Role and Scope
    • 4.2.1. Learning Your User Role(s)
  • 4.3. Managing Workloads
  • 4.4. Managing Environments
    • 4.4.1. Creating a New Environment
    • 4.4.2. Editing an Environment
  • 4.5. Managing Data Storage on the Cluster
    • 4.5.1. Data Sources
      • 4.5.1.1. PVC
        • 4.5.1.1.1. Creating a PVC
      • 4.5.1.2. Git
        • 4.5.1.2.1. Creating a Git Data Source
      • 4.5.1.3. ConfigMap
    • 4.5.2. Scratch/Ephemeral Storage
  • 4.6. Setting Up Credentials for Accessing Registries and Data
    • 4.6.1. Accessing Your NGC Org
    • 4.6.2. Setting Up Your NGC API Key
    • 4.6.3. Adding NGC Credentials to the Run:ai Cluster
    • 4.6.4. Adding Git Credentials
  • 4.7. Setting Up Templates
• 5. CLI/API Setup Guide
  • 5.1. Accessing the Run:ai CLI
    • 5.1.1. CLI Prerequisites
    • 5.1.2. Downloading the Run:ai CLI
    • 5.1.3. Setting up Your Kubernetes Configuration File
  • 5.2. Kubernetes Usage for Researchers
  • 5.3. Advanced Kubernetes Usage for Admins
  • 5.4. Setting up the Run:ai REST API on DGX Cloud
    • 5.4.1. References

Tutorials

• 1. Data Download Examples
  • 1.1. Download Data From S3 in a Training Workload
    • 1.1.1. Prerequisites and Requirements
    • 1.1.2. Creating the Credentials
    • 1.1.3. Creating the Data Source
    • 1.1.4. Creating the Training Workload
    • 1.1.5. Cleaning up the Environment
  • 1.2. Using Your Data From Google Cloud Storage
    • 1.2.1. Example: Using GCS to copy data to PVC within Run:ai
    • 1.2.2. Example: Using data directly within Google Cloud Services
    • 1.2.3. Example: Reading data from Google BigQuery
    • 1.2.4. Alternative to using Service Account Keys
  • 1.3. Using Your Data From Amazon S3
    • 1.3.1. Example: Using S3 to copy data to PVC within Run:ai
    • 1.3.2. Example: Using data directly within AWS Services
• 2. Interactive Workload Examples
  • 2.1. Interactive NeMo Workload Job
    • 2.1.1. Prerequisites and Requirements
    • 2.1.2. Creating the Data Source
    • 2.1.3. Creating the Workspace
    • 2.1.4. Running the Notebook
    • 2.1.5. Persisting the Model to Storage
    • 2.1.6. Cleaning up the Environment
  • 2.2. RAPIDS and Polars Workspace
    • 2.2.1. Prerequisites and Requirements
    • 2.2.2. Creating the Workspace
    • 2.2.3. Downloading the Polars Notebook
    • 2.2.4. Running the Polars Notebook
    • 2.2.5. Cleaning up the Environment
  • 2.3. Running Visual Studio Code Server
    • 2.3.1. Prerequisites and Requirements
    • 2.3.2. Creating the Workspace via CLI on a local workstation
    • 2.3.3. Cleaning up the Environment
  • 2.4. Building and Running an Interactive PyTorch Job with Visual Studio Code
    • 2.4.1. Prerequisites and Requirements
    • 2.4.2. Creating and Using a Custom Docker Container
    • 2.4.3. Run the Custom Container on DGX Cloud Using the Run:ai CLI
    • 2.4.4. Connect to the Running Workload Locally Using Visual Studio (via SSH)
    • 2.4.5. Cleaning up the Environment
  • 2.5. Using WandB with a Workspace
    • 2.5.1. Prerequisites and Requirements
    • 2.5.2. Creating the Credentials
    • 2.5.3. Creating the Workspace
    • 2.5.4. Connecting to WandB in your Jupyter Notebook
    • 2.5.5. Cleaning up the Environment
  • 2.6. Using BioNeMo Framework for ESM-2nv Data Preprocessing and Model Training
    • 2.6.1. Prerequisites and Requirements
    • 2.6.2. Creating the Data Source
    • 2.6.3. Creating an Interactive Workspace for Preprocessing
    • 2.6.4. Using Workspace to Preprocess Data for ESM-2nv
    • 2.6.5. Pretraining
      • 2.6.5.1. Single GPU Training in Jupyter Notebook
      • 2.6.5.2. Multi-Node Distributed Pretraining
• 3. Distributed Training Workload Examples
  • 3.1. Distributed PyTorch Training Job
    • 3.1.1. Requirements
    • 3.1.2. Creating a Custom Docker Container
    • 3.1.3. Creating a Data Source
    • 3.1.4. Creating the Workload via the UI
    • 3.1.5. Monitoring the Job
    • 3.1.6. Getting the Checkpoint
    • 3.1.7. Cleaning up the Environment
  • 3.2. Using Zero Quota Projects for Lower Priority Workloads
    • 3.2.1. Requirements
    • 3.2.2. Creating the Maximum Quota Project via the UI
    • 3.2.3. Creating the Zero Quota Project via the UI
    • 3.2.4. Creating the Lower Priority Interactive Workload via the UI
    • 3.2.5. Creating the Higher Priority Training Workload via the UI
    • 3.2.6. Cleaning up the Environment
• 4. NeMo End-to-End Workflow Example
  • 4.1. Requirements
  • 4.2. Initial Setup
    • 4.2.1. Create an Application
    • 4.2.2. Create a Persistent Volume Claim (PVC)
    • 4.2.3. Create a Hugging Face Account
    • 4.2.4. Accept the Llama-3.1-8B License
    • 4.2.5. Create a Weights & Biases Account
  • 4.3. Create an Interactive Visual Studio Code Job
    • 4.3.1. (Option 1) Connect to the VS Code Tunnel Locally
    • 4.3.2. (Option 2) Connect to the VS Code Tunnel Through a browser
  • 4.4. Prepare the Data
    • 4.4.1. Set Up the Script
    • 4.4.2. Launch Data Preparation
  • 4.5. Pre-Train the Model
    • 4.5.1. Set Up the Environment
    • 4.5.2. Launch the Pre-Training Job
    • 4.5.3. View the Project Dashboard on Weights & Biases
  • 4.6. Deploy the Model for Inference
    • 4.6.1. Send Requests to the Deployed Model
    • 4.6.2. Clean Up
• 5. (Deprecated) NeMo 1.0 End to End Workflow Example
  • 5.1. Requirements
  • 5.2. Initial setup
    • 5.2.1. Hugging Face Account Creation
    • 5.2.2. Accepting Mixtral-8x7B License
    • 5.2.3. Creating Weights & Biases Account
    • 5.2.4. Create a PVC
  • 5.3. Create Training Helper with JupyterLab
  • 5.4. Data Preparation
    • 5.4.1. Script Setup
    • 5.4.2. Launching Data Preparation
  • 5.5. Pre-Training
    • 5.5.1. Setup
  • 5.6. Inference Deployment
    • 5.6.1. Sending Requests to Deployed Model
    • 5.6.2. Cleaning Up
• 6. Inference Examples
  • 6.1. Using the UI
    • 6.1.1. Prerequisites and Requirements
    • 6.1.2. Create an Environment
    • 6.1.3. Optional: Create a PVC Datastore
    • 6.1.4. Create Workload
• 7. MLOps Integration Examples
  • 7.1. Argo Workflows Integration
    • 7.1.1. Prerequisites and Requirements
    • 7.1.2. Installing Argo Workflows CLI
    • 7.1.3. Configure Target Namespace
    • 7.1.4. Installing Argo Workflows (Admin Only)
    • 7.1.5. Basic Argo Workflows Usage
      • 7.1.5.1. Hello World
      • 7.1.5.2. Hello GPU
    • 7.1.6. Access Argo Workflows Web Interface
    • 7.1.7. Clean Up
    • 7.1.8. Next Steps
  • 7.2. Ray Cluster Integration
    • 7.2.1. Prerequisites and Requirements
    • 7.2.2. Deploy a KubeRay Operator
    • 7.2.3. Deploy a KubeRay Cluster
    • 7.2.4. Submit a Ray Job
      • 7.2.4.1. Execute a Ray job in the head pod
      • 7.2.4.2. Submit a Ray job to the Ray Cluster via ray job submission SDK
    • 7.2.5. Cleanup

Appendix

• AWS-Based DGX Cloud Cluster Configuration
  • Amazon Elastic Fabric Adapters
    • Taking Advantage of Auto-Mounted EFA
    • Troubleshooting
    • Opt-Out
  • (Optional) Private Access
    • AWS PrivateLink
      • Introduction
      • Assumptions
      • Configuration
        • Validation
        • PrivateLink Endpoints
        • DNS
      • Usage
      • Resources
        • Required Permissions
    • AWS Transit Gateway
      • Introduction
      • Assumptions
      • Getting Started
      • (Example) AWS Transit Gateway Private Access Deployment
• GCP-Based DGX Cloud Cluster Configuration
  • GCP TCPXO Networking
    • Automatic TCPXO Enablement
    • Troubleshooting
    • Opt-Out
• Common Issues and Debugging Tips
  • Common Issues and Resolutions
• Security Restrictions and Cluster Limitations
  • Security Restrictions for Kubernetes
  • Cluster Limitations
• Notices
  • Notice
  • Trademarks
  • Copyright
• Documentation Feedback