RAPIDS Accelerator for Apache Spark - User Guide

Overview

• Overview
  • Performance & Cost Benefits
  • Ease of Use
  • A Unified AI framework for ETL + ML/DL

Qualification Tool

• Overview
  • What The Qualification Tool Is
  • How To Run The Qualification Tool
• Quickstart
  • Install
    • Prerequisites
    • Install the CLI Package
  • Running the Tool
    • Example Commands
    • Command Options
    • Qualification Output
• Output Details
  • Tuning Output
  • Understanding the Qualification Tool Output
    • Entire App Report
    • Stages Report
    • Execs Report
    • Statistics Report
    • App Meta Data Report
    • MLFunctions Report
    • MLFunctions Total Duration Report
  • Output Format
    • Text and CSV files
      • Application Report Summary
      • Per SQL Query Report Summary
      • CSV Reports
        • Entire App Report
        • Per SQL Report
        • Stages Report
        • Execs Report
        • Status Report
        • Cluster Information Report
        • ML Functions Report
        • ML Functions Duration Report
        • Unsupported Operators Report
• Benchmark Environments

Getting Started

• Overview
  • RAPIDS Accelerator Compatibility Overview
  • Spark GPU Scheduling Overview
  • Spark workload qualification
  • Spark benchmark
• On-prem Cluster or Local Mode
  • Spark Deployment Methods
  • Apache Spark Setup for GPU
  • Install Spark
  • Download the RAPIDS Accelerator jar
  • Install the GPU Discovery Script
  • Local Mode
  • Spark Standalone Cluster
  • Running on YARN
    • YARN 3.1.3 with Isolation and GPU Scheduling Enabled
    • YARN 2.10 with Isolation and GPU Scheduling Enabled
    • YARN without Isolation
    • MIG GPU on YARN
      • YARN version 3.3.0+
      • YARN version 3.1.2 until 3.3.0
  • Running on Kubernetes
  • Configuration and Tuning
    • Example Command Running on YARN
  • Example Join Operation
  • Enabling RAPIDS Shuffle Manager
  • Advanced Configuration
  • Monitoring
  • Debugging
  • Out of GPU Memory
• AWS EMR
  • Leveraging RAPIDS Accelerator User Tools for Qualification and Bootstrap
  • Qualify CPU Workloads for GPU Acceleration
  • Configure and Launch AWS EMR with GPU Nodes
    • Launch an EMR Cluster using AWS CLI
    • Running the RAPIDS Accelerator User Tools Bootstrap for Optimal Cluster Spark Settings
    • Running an Example Join Operation Using Spark Shell
    • Submit Spark jobs to an EMR Cluster Accelerated by GPUs
    • Running GPU Accelerated Mortgage ETL Example using EMR Notebook
      • Create EMR Notebook and Connect to EMR GPU Cluster
      • Run Mortgage ETL PySpark Notebook on EMR GPU Cluster
• Databricks
  • Prerequisites
  • Limitations
  • Start a Databricks Cluster
  • Docker Container for Databricks
  • Import the GPU Mortgage Example Notebook
  • Hints
• GCP Dataproc
  • Create a Dataproc Cluster Accelerated by GPUs
    • Create a Dataproc Cluster Using T4s
    • Build a Custom Dataproc Image to Reduce Cluster Init Time
    • Create a Dataproc Cluster using MIG with A100s
    • Cluster Creation Troubleshooting
  • Run Python or Scala Spark Notebook on a Dataproc Cluster Accelerated by GPUs
  • Submit Spark jobs to a Dataproc Cluster Accelerated by GPUs
  • Diagnosing a GPU Cluster
  • Bootstrap GPU Cluster with Optimized Settings
  • Qualify CPU Workloads for GPU Acceleration
  • Tune Applications on GPU Cluster
• Dataproc Serverless
  • Accelerating Batch Workloads with GPUs on Dataproc Serverless:
  • Dataproc Serverless Runtime RAPIDS Version:
  • Creating a Serverless Batch Workload with RAPIDS Accelerator:
    • join-dataframes.py
    • Spark Batch Submission Command:
  • Logging Configuration for Serverless Workloads:
  • Cost:
  • Using Custom RAPIDS Accelerator JAR for Serverless Workloads
    • Build Commands:
    • Spark batch submission command with custom JAR:
• Azure Synapse Analytics
• Kubernetes
  • Prerequisites
  • Docker Image Preparation
  • Running Spark Applications in the Kubernetes Cluster
    • Submitting a Simple Test Job
    • Running an Interactive Spark Shell
    • Running PySpark in Client Mode
  • Running Spark Applications using Spark Operator
• Spark Workload Qualification
  • Qualification and Profiling tool
    • Requirements
    • How to use
  • Get the Explain Output
    • Using the Configuration Flag for Explain Only Mode
      • Requirements
      • Usage
    • How to use the Function Call
      • Requirements with Spark 3.X
      • Function Documentation
      • Usage
  • Run Spark applications with RAPIDS Accelerator on a GPU Spark Cluster
    • Requirements
    • How to use
• Oracle Cloud Infrastructure
• Spark3 GPU Configuration Guide on Yarn 3.2.1

Tuning

• Tuning Guide
  • Number of Executors
  • Number of Tasks per Executor
  • Pooled Memory
  • Pinned Memory
  • Spill Storage
  • Locality Wait
  • Number of Concurrent Tasks per GPU
  • Shuffle Partitions
  • Input Files
  • Input Files’ column order
  • Input Partition Size
    • Input Partition Size with DataSource API
    • Input Partition Size with Hive API
  • Input File Caching
  • Columnar Batch Size
    • File Reader Batch Size
  • Metrics
    • SQL
    • Task
    • Time taken on the GPU
      • Spilling
    • Time taken on the CPU
  • Window Operations
• Best Practices
  • Workload Qualification
  • Performance Tuning
  • How to handle GPU OOM issues
    • Reduce the number of concurrent tasks per GPU
    • Install CUDA 11.5 or above version
    • Identify that SQL, job and stage is involved in the error
    • Increase the number of tasks/partitions based on the type of the problematic stage
      • Table Scan Stage
      • Shuffle Stage
    • Reduce columnar batch size and file reader batch size
    • File an issue or ask a question on the GitHub repo

Profiling Tool

• Overview
  • What The Profiling Tool Is
  • Auto-Tuner Support
  • How To Run The Profiling Tool
  • Metrics Definitions
• Quickstart
  • Install
    • Prerequisites
    • Install the CLI Package
  • Running The Tool
    • Environment Variables
    • Command Options
    • Sample Commands
    • Profiling Output
• Output Details
  • Understanding The Profiling Output
    • A. Collect Information or Compare Information
    • B. Analysis
    • C. Health Check
    • D. Recommended Configuration
  • Generating Visualizations

Additional Functionality

• RAPIDS Accelerator for Apache Spark ML Library Integration
  • Existing ML Libraries
    • RMM
  • GPU accelerated ML Library development
    • ColumnarRdd
    • Examples of Spark ML Implementations leveraging ColumnarRdd
• RAPIDS Shuffle Manager
  • Example Configuration
  • Supported Spark Versions
  • Multi-Threaded Mode
  • UCX Mode
    • Building for ARM
    • System Setup
      • Baremetal
        • Rocky UCX RPM
    • Docker Containers
    • Validating UCX Environment
    • Spark App Configuration
      • Databricks
      • UCX Environment Variables
    • Fine Tuning
      • Bounce Buffers
      • Spillable Store
        • Shuffle Garbage Collection
      • Known Issues
• Apache Iceberg Support
• Delta Lake Support
  • Reading Delta Lake Tables
    • Data Queries
    • Metadata Queries
  • Writing Delta Lake Tables
    • Delta Lake Versions Supported For Write
    • Write Operations Supported
      • Automatic Optimization of Writes
    • RapidsDeltaWrite Node in Query Plans
  • Merging Into Delta Lake Tables
    • Limitations with DeltaTable merge API on non-Databricks Platforms
    • RapidsProcessDeltaMergeJoin Node in Query Plans
  • Delete Operations on Delta Lake Tables
    • num_affected_rows Difference with Databricks
  • Update Operations on Delta Lake Tables
• RAPIDS Accelerator File Cache
  • Limitations of the File Cache
  • Configuring the File Cache
  • Tuning File Cache Performance
    • Immutable Input Files

Appendixes

• Frequently Asked Questions
  • What versions of Apache Spark does the RAPIDS Accelerator for Apache Spark support?
  • Which distributions are supported?
  • What CUDA versions are supported?
  • What hardware is supported?
  • How can I check if the RAPIDS Accelerator is installed, and which version is running?
  • What parts of Apache Spark are accelerated?
  • Is the Spark Dataset API supported?
  • What is the road-map like?
  • How much faster will my query run?
  • What operators are best suited for the GPU?
  • Are there initialization costs?
  • How long does it take to translate a query to run on the GPU?
  • How can I tell what will run on the GPU and what won’t run on it?
  • Why does the plan for the GPU query look different from the CPU query?
  • Why does explain() show that the GPU will be used even after setting spark.rapids.sql.enabled to false?
  • How are failures handled?
  • How does the Spark scheduler decide what to do on the GPU vs the CPU?
  • Is Dynamic Partition Pruning (DPP) Supported?
  • Is Adaptive Query Execution (AQE) Supported?
  • Why does my query show as not on the GPU when Adaptive Query Execution is enabled?
  • Does the RAPIDS Shuffle Manager support External Shuffle Service (ESS)?
  • Are cache and persist supported?
  • Can I cache data into GPU memory?
  • Is PySpark supported?
  • Are the R APIs for Spark supported?
  • Are the Java APIs for Spark supported?
  • Are the Scala APIs for Spark supported?
  • Is the GPU needed on the driver? Are there any benefits to having a GPU on the driver?
  • Are table layout formats supported?
  • How many tasks can I run per executor? How many should I run per executor?
  • How are spark.executor.cores, spark.task.resource.gpu.amount, and spark.rapids.sql.concurrentGpuTasks related?
  • Why are multiple GPUs per executor not supported?
  • Why are multiple executors per GPU not supported?
  • Is Multi-Instance GPU (MIG) supported?
  • How can I run custom expressions/UDFs on the GPU?
    • RAPIDS Accelerated UDFs
    • Automatic Translation of Scala UDFs to Apache Spark Operations
    • Optimize a row-based UDF in a GPU operation
  • Why is the size of my output Parquet/ORC file different?
  • Why am I getting the error Failed to open the timezone file when reading files?
  • Why am I getting an error when trying to use pinned memory?
  • Why am I getting a buffer overflow error when using the KryoSerializer?
  • Why am I getting “Unable to acquire buffer” or “Trying to free an invalid buffer”?
  • Is speculative execution supported?
  • Why is my query in GPU mode slower than CPU mode?
  • Why is the Avro library not found by RAPIDS?
  • What is the default RMM pool allocator?
  • What is a RetryOOM or SplitAndRetryOOM exception?
  • Encryption Support
  • Can the RAPIDS Accelerator work with Spark on Ray (RayDP)?
  • Why I can not allocate entire Grace Hopper architecture GPU memory ?
  • I have more questions, where do I go?
• Qualification Tool - Jar Usage
  • Setting Up Environment
    • Prerequisites
    • Getting the Tools Jar
  • Deploying Tools Jar
    • Running the Qualification Tool Standalone on Spark Event Logs
      • Tuning Spark Properties For GPU Clusters
    • Running Using a Spark Listener
      • Configuration
      • Run the Spark Application
    • Running the Qualification Tool Inside a Running Spark Application Using the API
      • Modify Your Application Code To Call the APIs
      • Run the Spark application
  • Java CMD Samples
• Profiling Tool - Jar Usage
  • Setting Up Environment
    • Prerequisites
    • Getting the Tools Jar
  • Running Tools Jar
    • Profiling Tool Options
    • Tuning Spark Properties For GPU Clusters
    • Processing Spark Event Logs
    • Processing Driver Logs
  • Java CMD Samples
    • Collection Modes
• Examples
  • Benchmarks
  • Profiling Tool
    • CLI Samples
• Glossary
• Contact Us
  • Technical Support
  • Issues and Discussions