spark-rapids/user-guide/24.08.01/partials/tools-autotuner.html

Currently, the Auto-Tuner calculates a set of configurations that impact the performance of Apache Spark apps executing on GPU. Those calculations can leverage cluster information (for example, memory, cores, Spark default configurations) as well as information processed in the application event logs. The tool also will recommend settings for the application assuming that the job will be able to use all the cluster resources (CPU and GPU) when it’s running. The values loaded from the app logs have higher precedence than the default configs.

To run the Auto-Tuner, enable the auto-tuner flag and optionally pass a valid --worker-info <FILE_PATH>. The Auto-Tuner needs to learn the system properties of the worker nodes that run application code in the cluster. The argument FILE_PATH can either be local or remote file (that is, HDFS).

If the --worker-info argument isn’t supplied, then the Auto-Tuner will only recommend tuned settings based on the job event log and not on any cluster or worker information since that isn’t available.

 1system:
 2  numCores: 32
 3  memory: 212992MiB
 4  numWorkers: 5
 5gpu:
 6  memory: 15109MiB
 7  count: 4
 8  name: T4
 9softwareProperties:
10  spark.driver.maxResultSize: 7680m
11  spark.driver.memory: 15360m
12  spark.executor.cores: '8'
13  spark.executor.instances: '2'
14  spark.executor.memory: 47222m
15  spark.executorEnv.OPENBLAS_NUM_THREADS: '1'
16  spark.scheduler.mode: FAIR
17  spark.sql.cbo.enabled: 'true'
18  spark.ui.port: '0'
19  spark.yarn.am.memory: 640m