RAPIDS Accelerator for Apache Spark - User Guide

Overview

• Overview
  • Performance & Cost Benefits
  • Ease of Use
  • A Unified AI framework for ETL + ML/DL

Qualification Tool

• Overview
  • What The Qualification Tool Is
  • How To Run The Qualification Tool
• Quickstart
  • Install
    • Prerequisites
    • Install the CLI Package
  • Running the Tool
    • Environment Variables
    • Command Options
      • Cost-Savings
    • Sample Commands
    • Qualification Output
    • TCO Calculator
• Jar Usage
  • Setting Up Environment
    • Prerequisites
    • Getting the Tools Jar
  • Deploying Tools Jar
    • Running the Qualification Tool Standalone on Spark Event Logs
    • Running Using a Spark Listener
      • Configuration
      • Run the Spark Application
    • Running the Qualification Tool Inside a Running Spark Application Using the API
      • Modify Your Application Code To Call the APIs
      • Run the Spark application
• Output Details
  • Understanding the Qualification Tool Output
    • Detailed App Report
    • Stages report
    • Execs Report
    • MLFunctions Report
    • MLFunctions Total Duration Report
  • Output Formats
    • HTML Report
      • Application Recommendations Summary
      • App-Details View
      • Raw Data
      • Per-SQL Data
    • Text and CSV files
      • Application Report Summary
      • Per SQL Query Report Summary
      • Detailed App Report
        • Entire App Report
        • Per SQL Report
        • Stages Report
        • Execs Report

Getting Started

• Overview
  • RAPIDS Accelerator Compatibility Overview
  • Spark GPU Scheduling Overview
  • Spark workload qualification
  • Spark benchmark
• RAPIDS Accelerator with On-prem Cluster or Local Mode
  • Spark Deployment Methods
  • Apache Spark Setup for GPU
  • Install Spark
  • Download the RAPIDS Accelerator jar
  • Install the GPU Discovery Script
  • Local Mode
  • Spark Standalone Cluster
  • Running on YARN
    • YARN 3.1.3 with Isolation and GPU Scheduling Enabled
    • YARN 2.10 with Isolation and GPU Scheduling Enabled
    • YARN without Isolation
    • MIG GPU on YARN
      • YARN version 3.3.0+
      • YARN version 3.1.2 until 3.3.0
  • Running on Kubernetes
  • RAPIDS Accelerator Configuration and Tuning
    • Example Command Running on YARN
  • Example Join Operation
  • Enabling RAPIDS Shuffle Manager
  • Advanced Configuration
  • Monitoring
  • Debugging
  • Out of GPU Memory
• RAPIDS Accelerator on AWS EMR
  • Leveraging RAPIDS Accelerator User Tools for Qualification and Bootstrap
  • Qualify CPU Workloads for GPU Acceleration
  • Configure and Launch AWS EMR with GPU Nodes
    • Launch an EMR Cluster using AWS Console (GUI)
      • Step 1: EMR Release and Application Bundle Selection
      • Step 2: Hardware
      • Step 3: General Cluster Settings
      • Step 4: Edit Software Configuration
      • Step 5: Security
      • Finish Cluster Configuration
    • Launch an EMR Cluster using AWS CLI
    • Running the RAPIDS Accelerator User Tools Bootstrap for Optimal Cluster Spark Settings
    • Running an Example Join Operation Using Spark Shell
    • Submit Spark jobs to an EMR Cluster Accelerated by GPUs
    • Running GPU Accelerated Mortgage ETL Example using EMR Notebook
      • Create EMR Notebook and Connect to EMR GPU Cluster
      • Run Mortgage ETL PySpark Notebook on EMR GPU Cluster
• RAPIDS Accelerator on Databricks
  • Prerequisites
  • Limitations
  • Start a Databricks Cluster
  • RAPIDS Accelerator for Apache Spark Docker Container for Databricks
  • Import the GPU Mortgage Example Notebook
  • Hints
• RAPIDS Accelerator on GCP Dataproc
  • Create a Dataproc Cluster Accelerated by GPUs
    • Create a Dataproc Cluster Using T4s
    • Build a Custom Dataproc Image to Reduce Cluster Init Time
    • Create a Dataproc Cluster using MIG with A100s
    • Cluster Creation Troubleshooting
  • Run Python or Scala Spark Notebook on a Dataproc Cluster Accelerated by GPUs
  • Submit Spark jobs to a Dataproc Cluster Accelerated by GPUs
  • Diagnosing a GPU Cluster
  • Bootstrap GPU Cluster with Optimized Settings
  • Qualify CPU Workloads for GPU Acceleration
  • Tune Applications on GPU Cluster
• RAPIDS Accelerator on Dataproc Serverless
  • Accelerating Batch Workloads with GPUs on Dataproc Serverless:
  • Dataproc Serverless Runtime RAPIDS Version:
  • Creating a Serverless Batch Workload with RAPIDS Accelerator:
    • join-dataframes.py
    • Spark Batch Submission Command:
  • Logging Configuration for Serverless Workloads:
  • Cost:
  • Using Custom RAPIDS Accelerator JAR for Serverless Workloads
    • Build Commands:
    • Spark batch submission command with custom JAR:
• RAPIDS Accelerator on Azure Synapse Analytics
• RAPIDS and Kubernetes
  • Prerequisites
  • Docker Image Preparation
  • Running Spark Applications in the Kubernetes Cluster
    • Submitting a Simple Test Job
    • Running an Interactive Spark Shell
    • Running PySpark in Client Mode
  • Running Spark Applications using Spark Operator
• RAPIDS and Alluxio
  • Prerequisites
  • Alluxio setup
  • RAPIDS Configuration
  • Alluxio auto mount for AWS S3 buckets
  • Configure whether the disks used by Alluxio are fast
  • Alluxio Troubleshooting
  • Alluxio reliability
    • Dealing with Client side delays in response from master or workers
    • Worker server tunings to fail fast
    • The logs
    • Auto start Alluxio the master and workers
  • Alluxio limitations
    • Alluxio does not sync metadata from UFS(e.g. S3) by default
  • Alluxio metrics
    • Monitor Alluxio live metrics based on Alluxio Master Web
    • Monitor Alluxio live metrics based on Grafana with Prometheus
      • Config Prometheus when creating Databricks cluster
      • Install and start Grafana locally
      • Forward the Prometheus port 9090 to local port
      • Config local Grafana to monitor the live metrics
      • View a specific live Alluxio metrics in Prometheus Web UI
    • View Alluxio historic metrics based on Grafana with Prometheus
      • Install and start Prometheus locally
      • Restore historic metrics into Prometheus
      • Start Prometheus
      • View Alluxio historic metrics based on Grafana
      • View a specific historic Alluxio metrics in Prometheus Web UI
• Spark Workload Qualification
  • Qualification and Profiling tool
    • Requirements
    • How to use
  • Get the Explain Output
    • Using the Configuration Flag for Explain Only Mode
      • Requirements
      • Usage
    • How to use the Function Call
      • Requirements with Spark 3.X
      • Function Documentation
      • Usage
  • Run Spark applications with RAPIDS Accelerator on a GPU Spark Cluster
    • Requirements
    • How to use
• RAPIDS Accelerator on Oracle Cloud Infrastructure
• Spark3 GPU Configuration Guide on Yarn 3.2.1

Tuning

• Tuning Guide
  • Number of Executors
  • Number of Tasks per Executor
  • Pooled Memory
  • Pinned Memory
  • Spill Storage
  • Locality Wait
  • Number of Concurrent Tasks per GPU
  • Shuffle Partitions
  • Input Files
  • Input Files’ column order
  • Input Partition Size
    • Input Partition Size with DataSource API
    • Input Partition Size with Hive API
  • Input File Caching
  • Columnar Batch Size
    • File Reader Batch Size
  • Metrics
    • SQL
    • Task
    • Time taken on the GPU
      • Spilling
    • Time taken on the CPU
  • Window Operations
• Best Practices on the RAPIDS Accelerator for Apache Spark
  • Workload Qualification
  • Performance Tuning
  • How to handle GPU OOM issues
    • Reduce the number of concurrent tasks per GPU
    • Install CUDA 11.5 or above version
    • Identify which SQL, job and stage is involved in the error
    • Increase the number of tasks/partitions based on the type of the problematic stage
      • Table Scan Stage
      • Shuffle Stage
    • Reduce columnar batch size and file reader batch size
    • File an issue or ask a question on the GitHub repo

Profiling Tool

• Profiling Tool
  • How to use the Profiling tool
    • Prerequisites
    • Step 01-A: Download the tools jar
    • Step 01-B: Download the Apache Spark 3 distribution
    • Step 2 How to run the Profiling tool
  • Understanding Profiling tool detailed output and examples
    • A. Collect Information or Compare Information(if more than 1 event logs are as input and option –compare is specified)
    • B. Analysis
    • C. Health Check
    • D. Recommended Configuration
  • Generating Visualizations
  • Processing driver logs
  • Profiling tool options
  • Auto-Tuner support
  • Profiling tool metrics definitions

Additional Functionality

• RAPIDS Accelerator for Apache Spark ML Library Integration
  • Existing ML Libraries
    • RMM
  • GPU accelerated ML Library development
    • ColumnarRdd
    • Examples of Spark ML Implementations leveraging ColumnarRdd
• RAPIDS Shuffle Manager
  • Multi-Threaded Mode
  • UCX Mode
    • System Setup
      • Baremetal
        • Rocky UCX RPM
    • Docker Containers
    • Validating UCX Environment
    • Spark App Configuration
      • Databricks
      • UCX Environment Variables
    • Fine Tuning
      • Bounce Buffers
      • Spillable Store
        • Shuffle Garbage Collection
      • Known Issues
• Apache Iceberg Support
  • Apache Iceberg Versions
  • Reading Tables
    • Metadata Queries
    • Row-level Delete and Update Support
    • Schema Evolution
    • Data Formats
      • Parquet
      • ORC
      • Avro
    • Reader Split Size
  • Writing Tables
• Delta Lake Support
  • Reading Delta Lake Tables
    • Data Queries
    • Metadata Queries
  • Writing Delta Lake Tables
    • Delta Lake Versions Supported For Write
    • Write Operations Supported
      • Automatic Optimization of Writes
    • RapidsDeltaWrite Node in Query Plans
  • Merging Into Delta Lake Tables
    • Limitations with DeltaTable merge API on non-Databricks Platforms
    • RapidsProcessDeltaMergeJoin Node in Query Plans
  • Delete Operations on Delta Lake Tables
    • num_affected_rows Difference with Databricks
  • Update Operations on Delta Lake Tables
• RAPIDS Accelerator File Cache
  • Limitations of the File Cache
  • Configuring the File Cache
  • Tuning File Cache Performance
    • Immutable Input Files

Appendixes

• Frequently Asked Questions
  • What versions of Apache Spark does the RAPIDS Accelerator for Apache Spark support?
  • Which distributions are supported?
  • What CUDA versions are supported?
  • What hardware is supported?
  • How can I check if the RAPIDS Accelerator is installed and which version is running?
  • What parts of Apache Spark are accelerated?
  • Is the Spark Dataset API supported?
  • What is the road-map like?
  • How much faster will my query run?
  • What operators are best suited for the GPU?
  • Are there initialization costs?
  • How long does it take to translate a query to run on the GPU?
  • How can I tell what will run on the GPU and what will not run on it?
  • Why does the plan for the GPU query look different from the CPU query?
  • Why does explain() show that the GPU will be used even after setting spark.rapids.sql.enabled to false?
  • How are failures handled?
  • How does the Spark scheduler decide what to do on the GPU vs the CPU?
  • Is Dynamic Partition Pruning (DPP) Supported?
  • Is Adaptive Query Execution (AQE) Supported?
  • Why does my query show as not on the GPU when Adaptive Query Execution is enabled?
  • Does the RAPIDS Shuffle Manager support External Shuffle Service (ESS)?
  • Are cache and persist supported?
  • Can I cache data into GPU memory?
  • Is PySpark supported?
  • Are the R APIs for Spark supported?
  • Are the Java APIs for Spark supported?
  • Are the Scala APIs for Spark supported?
  • Is the GPU needed on the driver? Are there any benefits to having a GPU on the driver?
  • Are table layout formats supported?
  • How many tasks can I run per executor? How many should I run per executor?
  • How are spark.executor.cores, spark.task.resource.gpu.amount, and spark.rapids.sql.concurrentGpuTasks related?
  • Why are multiple GPUs per executor not supported?
  • Why are multiple executors per GPU not supported?
  • Is Multi-Instance GPU (MIG) supported?
  • How can I run custom expressions/UDFs on the GPU?
    • RAPIDS Accelerated UDFs
    • Automatic Translation of Scala UDFs to Apache Spark Operations
    • Optimize a row-based UDF in a GPU operation
  • Why is the size of my output Parquet/ORC file different?
  • Why am I getting the error Failed to open the timezone file when reading files?
  • Why am I getting an error when trying to use pinned memory?
  • Why am I getting a buffer overflow error when using the KryoSerializer?
  • Why am I getting “Unable to acquire buffer” or “Trying to free an invalid buffer”?
  • Is speculative execution supported?
  • Why is my query in GPU mode slower than CPU mode?
  • Why is the Avro library not found by RAPIDS?
  • What is the default RMM pool allocator?
  • What is a RetryOOM or SplitAndRetryOOM exception?
  • Encryption Support
  • Can the RAPIDS Accelerator work with Spark on Ray (RayDP)?
  • I have more questions, where do I go?
• Examples
  • Benchmarks
  • Qualification Tool
    • CLI Samples
    • Java CMD Samples
• Glossary
• Contact Us
  • Technical Support
  • Issues and Discussions