Is this page helpful?

Overview#

What The Profiling Tool Is#

The Profiling tool analyzes both CPU or GPU-generated event logs and generates information that can be used for debugging and profiling Apache Spark applications.

In addition, the Profiling tool can process GPU a driver log to list any unsupported operators.

The output information contains the Spark version, executor details, properties, etc. The tool also will recommend setting for the application assuming that the job will be able to use all the cluster resources (CPU and GPU) when it’s running. The Profiling tool optionally provides optimized RAPIDS configurations based on the worker’s information (refer to Auto-Tuner Support).

Auto-Tuner Support#

Auto-Tuner aims at optimizing Apache Spark applications by recommending a set of configurations to tune the performance of Rapids accelerator.

Currently, the Auto-Tuner calculates a set of configurations that impact the performance of Apache Spark apps executing on GPU. Those calculations can leverage cluster information (for example, memory, cores, Spark default configurations) as well as information processed in the application event logs. The tool also will recommend settings for the application assuming that the job will be able to use all the cluster resources (CPU and GPU) when it’s running.

Note

Auto-Tuner requirements:

  • It runs only on GPU event logs — that is, event logs produced by Spark applications that ran with the RAPIDS Accelerator plugin enabled.

  • For the jar entry point, enable the module by passing --auto-tuner.

How To Run The Profiling Tool#

RAPIDS Accelerator for Apache Spark CLI tool

The simplest way to run the Profiling tool. In running the Profiling tool standalone on Spark event logs, the tool can be run as a user tool command via a pip package for CSP environments (Google Dataproc, AWS EMR, Databricks-AWS, and Databricks-Azure) in addition to on-prem.

The tool will output the profile information per application along with recommended GPU cluster shape and RAPIDS configurations. Additionally, it provides a set of tuning recommendations specifically tailored for Spark applications running on GPU clusters, as part of the default output from the Auto-Tuner feature. For more information on running the Profiling tool from the pip-package, visit the quick start guide

Java API

The java API can be used for other environments that aren’t supported by the CLI tool. The tool processes each event log individually and writes a per-application summary text file named profile.log under “rapids_4_spark_profile/{APP_ID}”, along with the reports described in the output details section. When --auto-tuner is passed and the event log is from a GPU run, the tool also emits recommended Spark configurations — see Tuning Spark Properties For GPU Clusters.

Metrics Definitions#

The Profiling tool generates some SQL metrics that can be found in the following docs: