Quickstart

spark-rapids-user-tools CLI enables user to run the tool for logs from a number of CSP platforms in addition to on-prem.

Prerequisites

• Set up a Python environment with a version between 3.8 and 3.11

• Java 8+

• The developer machine used to host the CLI needs internet access. If the machine is behind a proxy, it’s recommended to install the CLI package from source using fat mode as described in Install the CLI Package.

• Set up the development environment for your CSP or on-prem

  • On-prem
  • Dataproc
  • EMR
  • Databricks-AWS
  • Databricks-Azure
  • Local filesystem
  • HDFS

  No more steps required to run the tools on on-premises environment including standalone/local machines.

  The tools CLI depends on Python implementation of PyArrow which relies on some environment variables to bind with HDFS:

  • HADOOP_HOME: the root of your installed Hadoop distribution. Often has “lib/native/libhdfs.so”.

  • JAVA_HOME: the location of your Java SDK installation.

  • ARROW_LIBHDFS_DIR (optional): explicit location of “libhdfs.so” if it’s installed somewhere other than $HADOOP_HOME/lib/native.

  • Add the Hadoop jars to your CLASSPATH.

    • Linux
    • Windows
    Copy

    Copied!

                
    
                
    export CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath --glob`
            

    Copy

    Copied!

                
    
                
    %HADOOP_HOME%/bin/hadoop classpath --glob > %CLASSPATH%
            

  For more information on HDFS requirements, refer to the PyArrow HDFS documentation

  
  

  • Install gcloud CLI. Follow the instructions on gcloud-sdk-install

  • Set the configuration settings and credentials of the gcloud CLI:

    • Initialize the gcloud CLI by following these instructions

    • Grant authorization to the gcloud CLI with a user account

    • Set up “application default credentials” to the gcloud CLI by logging in

    • Manage gcloud CLI configurations. For more details, visit gcloud-sdk-configurations

    • Verify that the following gcloud CLI properties are properly defined:

      • dataproc/region

      • compute/zone

      • compute/region

      • core/project

    • If the configuration isn’t set to default values, then make sure to explicitly set some environment variables to be picked up by the tools cmd such as: CLOUDSDK_DATAPROC_REGION, and CLOUDSDK_COMPUTE_REGION.

    • The tools CLI follows the process described in this doc to resolve the credentials. If not running on (GCP), the environment variable GOOGLE_APPLICATION_CREDENTIALS is required to point to a JSON file containing credentials.

  • Install the AWS CLI version 2. Follow the instructions on aws-cli-getting-started

  • Set the configuration settings and credentials of the AWS CLI by creating credentials and config files as described in aws-cli-configure-files.

  • If the AWS CLI configuration isn’t set to the default values, then make sure to explicitly set some environment variables tp be picked up by the tools cmd such as: AWS_PROFILE, AWS_DEFAULT_REGION, AWS_CONFIG_FILE, AWS_SHARED_CREDENTIALS_FILE. Refer to the full list of variables in aws-cli-configure-envvars

  • It’s important to configure with the correct region for the bucket being used on S3. If region isn’t set, the AWS SDK will choose a default value that may not be valid. In addition, the tools CLI by inspects AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY emvironment variables if the credentials couldn’t be pulled from the credential files.

  Note

  In order to be able to run tools that require SSH on the EMR nodes (that is, bootstrap), then:

  • make sure that you have SSH access to the cluster nodes; and

  • create a key pair using Amazon EC2 through the AWS CLI command aws ec2 create-key-pair as instructed in aws-cli-create-key-pairs.

  The tool currently only supports event logs stored on S3 (no DBFS paths). The remote output storage is also expected to be S3. In order to get complete eventlogs for a given run-id : ` databricks clusters list | grep <run-id> databricks fs cp -r <databricks log location/<cluster id got from the above command> <destination_location> ` are a couple of commands that can be used to download all the logs associated with a given run. Please refer to the latest Databricks documentation on up-to-date information. Due to some platform limitations, it is likely that the logs may be incomplete. Thq qualification tool attempts to process them as best as possible. If the results come back empty, the rapids_4_spark_qualification_output_status.csv file can call out the failed run due to incomplete logs.

  • Install Databricks CLI

    • Install the Databricks CLI version 0.200+. Follow the instructions on Install the CLI.

    • Set the configuration settings and credentials of the Databricks CLI:

    • Set up authentication by following these instructions

    • Verify that the access credentials are stored in the file ~/.databrickscfg on Unix, Linux, or macOS, or in another file defined by environment variable DATABRICKS_CONFIG_FILE.

    • If the configuration isn’t set to default values, then make sure to explicitly set some environment variables to be picked up by the tools cmd such as: DATABRICKS_CONFIG_FILE, DATABRICKS_HOST and DATABRICKS_TOKEN. Refer to the description of the variables in environment variables docs.

  • Setup the environment to access S3

    • Install the AWS CLI version 2. Follow the instructions on aws-cli-getting-started

    • Set the configuration settings and credentials of the AWS CLI by creating credentials and config files as described in aws-cli-configure-files.

    • If the AWS CLI configuration isn’t set to the default values, then make sure to explicitly set some environment variables tp be picked up by the tools cmd such as: AWS_PROFILE, AWS_DEFAULT_REGION, AWS_CONFIG_FILE, AWS_SHARED_CREDENTIALS_FILE. Refer to the full list of variables in aws-cli-configure-envvars

    • It’s important to configure with the correct region for the bucket being used on S3. If region isn’t set, the AWS SDK will choose a default value that may not be valid. In addition, the tools CLI by inspects AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY emvironment variables if the credentials couldn’t be pulled from the credential files.

    Note

    In order to be able to run tools that require SSH on the EMR nodes (that is, bootstrap), then:

    • make sure that you have SSH access to the cluster nodes; and

    • create a key pair using Amazon EC2 through the AWS CLI command aws ec2 create-key-pair as instructed in aws-cli-create-key-pairs.

  The tool currently only supports event logs stored on ABFS. The remote output storage is also expected to be ABFS (no DBFS paths). In order to get complete eventlogs for a given run-id : ` databricks clusters list | grep <run-id> databricks fs cp -r <databricks log location/<cluster id got from the above command> <destination_location> ` are a couple of commands that can be used to download all the logs associated with a given run. Please refer to the latest Databricks documentation on up-to-date information. Due to some platform limitations, it is likely that the logs may be incomplete. Thq qualification tool attempts to process them as best as possible. If the results come back empty, the rapids_4_spark_qualification_output_status.csv file can call out the failed run due to incomplete logs.

  • Install Databricks CLI

    • Install the Databricks CLI version 0.200+. Follow the instructions on Install the CLI.

    • Set the configuration settings and credentials of the Databricks CLI:

    • Set up authentication by following these instructions

    • Verify that the access credentials are stored in the file ~/.databrickscfg on Unix, Linux, or macOS, or in another file defined by environment variable DATABRICKS_CONFIG_FILE.

    • If the configuration isn’t set to default values, then make sure to explicitly set some environment variables to be picked up by the tools cmd such as: DATABRICKS_CONFIG_FILE, DATABRICKS_HOST and DATABRICKS_TOKEN. Refer to the description of the variables in environment variables docs.

  • Install Azure CLI

    • Install the Azure CLI. Follow the instructions on How to install the Azure CLI.

    • Set the configuration settings and credentials of the Azure CLI:

      • Set up the authentication by following these instructions.

      • Configure the Azure CLI by following these instructions.

        • location is used for retreving instance type description (default is westus).

        • output should use default of json in core section.

        • Verify that the configurations are stored in the file $AZURE_CONFIG_DIR/config where the default value of AZURE_CONFIG_DIR is $HOME/.azure on Linux or macOS.

    • If the configuration isn’t set to default values, then make sure to explicitly set some environment variables to be picked up by the tools cmd such as: AZURE_CONFIG_DIR and AZURE_DEFAULTS_LOCATION.

Install the CLI Package

            

            
pip install spark-rapids-user-tools
        

            

            
pip install <wheel-file>
        

A typical workflow to successfully run the qualification command:

1. Follow the instructions to set up the prerequisites and install the CLI

2. Get Apache Spark eventlogs from prior runs of CPU based applications on Spark 2.x or later. In addition to local storage, the eventlogs should be stored in a valid remote storage:

   • For Dataproc, it should be set to GCS path.

   • For EMR and Databricks-AWS, it should be set to S3 path.

   • For Databricks-Azure, it should be set to ABFS path.

3. Run the qualification command on the set of selected eventlogs. Event logs can be passed as single files, a directory, a comma-separated list of files or directories. The format of event logs can be raw, zip, or gzip.

            

            
spark_rapids qualification <flags>
        

The tool helps quantify the expected acceleration of migrating a Spark application or query to GPU. The tool will process each app individually, but will group apps with the same name and cluster details into a single output row after averaging duration metrics accordingly.

Example Commands

This section shows examples of Qualification CLI commands assuming the following inputs:

• EVENTLOG: Path to Spark eventlogs without the scheme part. The scheme can be a local file system (file://), HDFS (hdfs://), S3 (s3://), ABFS (abfss://), or GCS (gs://).

The following table shows CLI command examples along with platform and expected functionalities based on which analysis is performed.

            

            
spark_rapids qualification \
  --platform dataproc \
  --eventlogs gs://$EVENTLOG
        

            

            
spark_rapids qualification \
  --platform emr \
  --eventlogs s3://$EVENTLOG
        

            

            
spark_rapids qualification \
  --platform databricks-azure \
  --eventlogs file://$EVENTLOG
        

            

            
spark_rapids qualification \
  --platform databricks-aws \
  --eventlogs file://$EVENTLOG
        

            

            
spark_rapids qualification \
  --platform onprem \
  --eventlogs file://$EVENTLOG
        

Command Options

You can list all the options using the help argument

            

            
spark_rapids qualification -- --help
        

Available options are listed in the following table.

Cluster-Metadata

By default, the qualification tool generates estimated speedups of the CPU applications. It will also generate a cluster recommendation for running on GPU. To aid that recommendation, you can provide the CPU cluster information.

The specific type of parameter passed for the cluster is based on platform, see the following scenarios to determine which method to use for your platform:

            

            
spark_rapids qualification --cluster $CLUSTER_NAME [flags]
        

            

            
spark_rapids qualification --cluster $CLUSTER_ID [flags]
        

            

            
spark_rapids qualification --cluster $CLUSTER_PROPS [flags]
        

            

            
config:
  masterConfig:
    numCores: 2
    memory: 7680MiB
  workerConfig:
    numCores: 8
    memory: 7680MiB
    numWorkers: 2
        

            

            
gcloud dataproc clusters describe $CLUSTER_NAME > $CLUSTER_PROPS
        

            

            
aws emr describe-cluster --cluster-id $CLUSTER_ID > $CLUSTER_PROPS
        

            

            
databricks clusters get $CLUSTER_ID > $CLUSTER_PROPS
        

            

            
databricks clusters get $CLUSTER_ID > $CLUSTER_PROPS
        

Qualification Output

The Qualification tool will run against logs from your CSP environment and then will output the applications recommended for acceleration along with Estimated GPU Speedup.

The command creates a directory with UUID that contains the following output:

            

            
  qual_20230314145334_d2CaFA34
  ├── app_metadata.json
  ├── qualification_summary.csv
  ├── qualification_statistics.csv
  ├── intermediate_output/
  ├── rapids_4_spark_qualification_output/
  └── xgboost_predictions/
      ...
        

In qualification_summary.csv, the command output lists these key fields for each application:

App ID

An application is referenced by its application ID, app-id. When running on YARN, each application may have multiple attempts, but there are attempt IDs only for applications in cluster mode, not applications in client mode. Applications in YARN cluster mode can be identified by their attempt-id.

App Name

Name of the application

App Duration

Wall-Clock time measured since the application starts until it’s completed. If an app isn’t completed an estimated completion time would be computed.

Estimated GPU Duration

Predicted runtime of the app if it was run on GPU.
It’s the sum of the accelerated operator durations and ML functions duration(if applicable) along with durations that couldn’t run on GPU because they’re unsupported operators or not SQL/Dataframe.

Estimated GPU Speedup

That will estimate how much faster the application would run on GPU. It’s calculated as the ratio between App Duration and Estimated GPU Duration.

Estimated GPU Speedup Category

This is the qualification result per job to determine if the job is a good candidate for running on GPU. A value of Large, Medium, or Small indicates the job should be migrated to GPU with different levels of confidence for expected acceleration. A value of Not Recommended or Not Applicable indicates the job should not be migrated to GPU.

The qualification command output in STDOUT will also show a summary of qualified candidates.

Sample of Qualification command output in STDOUT

            

            
  +----+-----------------+--------------------------------+-----------------+------------------+-------------------------------------+------------------------------------+
  |    | App Name        | App ID                         | Estimated GPU   |   Qualified Node | Full Cluster                        | GPU Config                         |
  |    |                 |                                | Speedup         |   Recommendation | Config                              | Recommendation                     |
  |    |                 |                                | Category**      |                  | Recommendations*                    | Breakdown*                         |
  |----+-----------------+--------------------------------+-----------------+------------------+-------------------------------------+------------------------------------|
  |  0 | query A0        | application_1696859475058_0007 | Large           | 45 x g5.8xlarge  | application_1696859475058_0007.conf | application_1696859475058_0007.log |
  |  1 | query A1        | application_1696859475058_0008 | Large           | 9 x g5.8xlarge   | application_1696859475058_0008.conf | application_1696859475058_0008.log |
  |  2 | query A2        | application_1696859475058_0012 | Large           | 2 x g5.8xlarge   | application_1696859475058_0012.conf | application_1696859475058_0012.log |
  |  5 | query B0        | application_1696859475058_0014 | Medium          | 150 x g5.8xlarge | application_1696859475058_0014.conf | application_1696859475058_0014.log |
  |  6 | query B1        | application_1696859475058_0016 | Medium          | 2 x g5.8xlarge   | application_1696859475058_0016.conf | application_1696859475058_0016.log |
  |  7 | query B2        | application_1696859475058_0011 | Small           | 2 x g5.8xlarge   | application_1696859475058_0011.conf | application_1696859475058_0011.log |
  +----+-----------------+--------------------------------+-----------------+------------------+-------------------------------------+------------------------------------+

  Notes:
  --------------------
   - *Cluster config recommendations: ./qual_20240731164401_6D6B7fb8/rapids_4_spark_qualification_output/tuning
   - **Estimated GPU Speedup Category assumes the user is using the node type recommended and config recommendations with the same size cluster as was used with the CPU side.
        

For more information on the detailed output of the Qualification tool, go here: Output Details.