Output Details#
The qualification tool generates a number of detailed reports in addition to the high-level recommendations. The summary report goes to STDOUT and the full set of files is written under a timestamped run directory (for example, qual_20240814145334_d2CaFA34). The output goes into your default filesystem and supports both local filesystem and HDFS. If you are on an HDFS cluster the default filesystem is likely HDFS for both the input and output; to point to the local filesystem, include the file: prefix in the path.
Inside the run directory, the output is organised in three tiers:
Top-level wrapper outputs —
qualification_summary.csv,qualification_statistics.csv,app_metadata.json, QualX predictions underxgboost_predictions/, and the nestedqual_core_output/subdirectory produced by the core jar.Core global outputs under
qual_core_output/—apps_summary.csv,status.csv, andruntime.properties.Per-application outputs under
qual_core_output/qual_metrics/<app-id>/(qualification metrics) andqual_core_output/tuning_apps/<app-id>/(AutoTuner recommendations, when--auto-tuneris enabled).
The tree structure of the output directory is as follows:
1 qual_20240814145334_d2CaFA34
2 ├── app_metadata.json
3 ├── qual_core_stderr.log
4 ├── qualification_statistics.csv
5 ├── qualification_summary.csv
6 ├── intermediate_output
7 │ └── heuristics_info.csv
8 ├── qual_core_output
9 │ ├── apps_summary.csv
10 │ ├── runtime.properties
11 │ ├── status.csv
12 │ ├── qual_metrics
13 │ │ ├── app-001-0001
14 │ │ │ ├── cluster_information.json
15 │ │ │ ├── execs.csv
16 │ │ │ ├── operators_stats.csv
17 │ │ │ ├── persql.csv
18 │ │ │ ├── stages.csv
19 │ │ │ ├── udf_report.json
20 │ │ │ └── unsupported_operators.csv
21 │ │ └── app-002-0001
22 │ │ ├── cluster_information.json
23 │ │ ├── execs.csv
24 │ │ ├── operators_stats.csv
25 │ │ ├── persql.csv
26 │ │ ├── stages.csv
27 │ │ ├── udf_report.json
28 │ │ └── unsupported_operators.csv
29 │ ├── tuning_apps
30 │ │ ├── app-001-0001
31 │ │ │ ├── bootstrap.conf
32 │ │ │ ├── combined.conf
33 │ │ │ └── recommendations.log
34 │ │ └── app-002-0001
35 │ │ ├── bootstrap.conf
36 │ │ ├── combined.conf
37 │ │ └── recommendations.log
38 │ ├── tuning
39 │ │ ├── app-001-0001-bootstrap.conf
40 │ │ ├── app-001-0001.conf
41 │ │ ├── app-001-0001.log
42 │ │ ├── app-002-0001-bootstrap.conf
43 │ │ ├── app-002-0001.conf
44 │ │ └── app-002-0001.log
45 │ └── raw_metrics
46 │ ├── app-001-0001
47 │ │ ├── application_information.csv
48 │ │ ├── application_log_path_mapping.csv
49 │ │ ├── data_source_information.csv
50 │ │ ├── executor_information.csv
51 │ │ ├── io_metrics.csv
52 │ │ ├── job_information.csv
53 │ │ ├── job_level_aggregated_task_metrics.csv
54 │ │ ├── profile.log
55 │ │ ├── spark_properties.csv
56 │ │ ├── sql_duration_and_executor_cpu_time_percent.csv
57 │ │ ├── sql_level_aggregated_task_metrics.csv
58 │ │ ├── sql_plan_info_pre_aqe.json
59 │ │ ├── sql_plan_metrics_for_application.csv
60 │ │ ├── sql_to_stage_information.csv
61 │ │ ├── stage_level_aggregated_task_metrics.csv
62 │ │ ├── stage_level_all_metrics.csv
63 │ │ ├── system_properties.csv
64 │ │ ├── wholestagecodegen_mapping.csv
65 │ │ └── write_operations.csv
66 │ └── app-002-0001
67 │ ├── application_information.csv
68 │ ├── application_log_path_mapping.csv
69 │ ├── data_source_information.csv
70 │ ├── executor_information.csv
71 │ ├── io_metrics.csv
72 │ ├── job_information.csv
73 │ ├── job_level_aggregated_task_metrics.csv
74 │ ├── profile.log
75 │ ├── spark_properties.csv
76 │ ├── sql_duration_and_executor_cpu_time_percent.csv
77 │ ├── sql_level_aggregated_task_metrics.csv
78 │ ├── sql_plan_info_pre_aqe.json
79 │ ├── sql_plan_metrics_for_application.csv
80 │ ├── sql_to_stage_information.csv
81 │ ├── stage_level_aggregated_task_metrics.csv
82 │ ├── stage_level_all_metrics.csv
83 │ ├── system_properties.csv
84 │ ├── wholestagecodegen_mapping.csv
85 │ └── write_operations.csv
86 └── xgboost_predictions
87 ├── feature_importance.csv
88 ├── features.csv
89 ├── per_app.csv
90 ├── per_sql.csv
91 └── shap_values.csv
For information on the files content and processing the Qualification report and the recommendation, refer to Understanding the Qualification tool output and Output Formats sections below.
Tuning Output#
When the --auto-tuner argument is enabled, AutoTuner results are written under qual_core_output/tuning_apps/<app-id>/ with three files per application:
bootstrap.conf: the minimum set of Spark configurations required to enable GPU execution for the application.recommendations.log: the full tuning recommendations with accompanying comments.combined.conf: the combined Spark properties (bootstrap + tuned values) ready to pass tospark-submit.
A legacy flat layout is also written under qual_core_output/tuning/ (<app-id>-bootstrap.conf, <app-id>.log, <app-id>.conf) for backward compatibility with consumers that still read the older paths.
Certain properties with sensitive information are redacted in the combined results.
The Auto-Tuner output has 2 main sections:
Spark Properties: A list of Apache Spark configurations to tune the performance of the app. The list is the result of
diffbetween the existing app configurations and the recommended ones. Therefore, a recommendation matches the existing app configuration, it won’t show up in the list.Comments: A list of messages to highlight properties that were missing in the app configurations, or the cause of failure to generate the recommendations.
Examples
1 Spark Properties:
2
3 –conf spark.executor.cores=16
4 –conf spark.executor.instances=8
5 –conf spark.executor.memory=32768m
6 –conf spark.executor.memoryOverhead=7372m
7 –conf spark.rapids.memory.pinnedPool.size=4096m
8 –conf spark.rapids.sql.concurrentGpuTasks=2
9 –conf spark.sql.files.maxPartitionBytes=512m
10 –conf spark.sql.shuffle.partitions=200
11 –conf spark.task.resource.gpu.amount=0.0625
12
13Comments:
14
15- ‘spark.executor.instances’ wasn't set.
16- ‘spark.executor.cores’ wasn't set.
17- ‘spark.task.resource.gpu.amount’ wasn't set.
18- ‘spark.rapids.sql.concurrentGpuTasks’ wasn't set.
19- ‘spark.executor.memory’ wasn't set.
20- ‘spark.rapids.memory.pinnedPool.size’ wasn't set.
21- ‘spark.executor.memoryOverhead’ wasn't set.
22- ‘spark.sql.files.maxPartitionBytes’ wasn't set.
23- ‘spark.sql.shuffle.partitions’ wasn't set.
24- ‘spark.sql.adaptive.enabled’ should be enabled for better
25 performance.
1Spark Properties:
2
3--conf spark.executor.instances=8
4--conf spark.sql.shuffle.partitions=200
5
6Comments:
7
8- 'spark.sql.shuffle.partitions' wasn't set.
1Cannot recommend properties. Refer to Comments.
2
3Comments:
4- java.io.FileNotFoundException: File worker-info.yaml doesn't exist
5- 'spark.executor.memory' should be set to at least 2GB/core.
6- 'spark.executor.instances' should be set to (gpuCount * numWorkers).
7- 'spark.task.resource.gpu.amount' should be set to Max(1, (numCores / gpuCount)).
8- 'spark.rapids.sql.concurrentGpuTasks' should be set to Min(4, (gpuMemory / 7.5G)).
9- 'spark.rapids.memory.pinnedPool.size' should be set to 2048m.
10- 'spark.rapids.sql.enabled' should be true to enable SQL operations on the GPU.
11- 'spark.sql.adaptive.enabled' should be enabled for better performance.
Understanding the Qualification Tool Output#
For each processed Spark application, the Qualification tool generates three main fields to help quantify the expected acceleration of migrating a Spark application or query to GPU.
Estimated GPU Speedup Category: summary recommendation for the application for migration to GPU. Can be Large, Medium, Small, or Not Recommended. The size (Large, Medium, or Small) represents the likelihood of acceleration on GPU for an application that is qualified. A result of Not Recommended indicates the application is not a candidate for migration to GPU.
Estimated Speedup: the estimated speed-up is simply the original CPU duration of the app divided by the estimated GPU duration. That will estimate how much faster the application would run on GPU.
Estimated GPU Duration: predicted runtime of the app if it was run on GPU. It is the sum of the accelerated operator durations and ML functions duration(if applicable) along with durations that could not run on GPU because they are unsupported operators or not SQL/Dataframe.
The lower the estimated GPU duration, the higher the “Estimated Speed-up”. The processed applications or queries are ranked by the “Estimated Speed-up”.
As mentioned before, the tool doesn’t guarantee the applications or queries with the highest recommendation will actually be accelerated the most. Refer to Supported operators guide.
In addition to the recommendation, the Qualification tool reports a set of metrics in tasks of SQL Dataframe operations within the scope of: “Entire App”; “Stages”; and “Execs.” The report is divided into three main levels. The fields of each level are described in details in the following sections: Entire App Report, Stages report, and Execs report. Then we describe the output formats and their file locations in Output Formats section.
There’s an option --per-sql to print a report at the SQL query level in addition to the application level.
Entire App Report#
This report contains one row per successfully-processed application and is saved as qual_core_output/apps_summary.csv. The QualX-derived fields (Estimated GPU Duration, Estimated GPU Speedup, Estimated GPU Speedup Category, Recommendation, GPU Opportunity, Unsupported Execs / Expressions) are merged in by the CLI wrapper and appear in the top-level qualification_summary.csv and qualification_statistics.csv files.
Columns in apps_summary.csv:
Attempt ID: attempt ID of the app associated with the eventlog. For YARN, the unique ID of the app is the concatenation of the application ID and the attempt ID.
Executor CPU Time Percent: this is an estimate at how much time the tasks spent doing processing on the CPU versus waiting on IO. This isn’t always a good indicator because sometimes the IO is encrypted and the CPU has to do work to decrypt it, so the environment you are running on needs to be taken into account.
SQL Ids with Failures: comma-separated list of SQL Ids of queries with failed jobs.
Unsupported Read File Formats and Types: looks at the Read Schema and reports the file formats along with types that may not be fully supported. Example:
JDBC[*]. This is based on the current version of the plugin and future versions may add support for more file formats and types.Unsupported Write Data Format: reports the data format that we currently don’t support, that is, if the result is written in JSON or CSV format.
Complex Types: looks at the Read Schema and reports if there are any complex types (array, struct, or maps) in the schema.
Nested Complex Types: nested complex types are complex types that contain other complex types (Example:
array<struct<string,string>>). It can read all the schemas for DataSource V1. The Data Source V2 truncates the schema, so if you see “...”, then the full schema isn’t available. For such schemas we read until...and report if there are any complex types and nested complex types in that.Potential Problems: some UDFs and nested complex types. Keep in mind that the tool is only able to detect certain issues.
Longest SQL Duration: the maximum amount of time spent in a single task of SQL Dataframe operations.
SQL Stage Durations Sum: the total wall-clock duration (
completionTime - submissionTime) of all the stages in the app.App Duration Estimated: True or False indicates if the application duration had to be estimated. If True, the eventlog was missing the application finished event, so the last job or SQL execution time was used as the end time.
Total Core Seconds: sums the execution time for all executors multiplied by the number of cores in each executor to get total core seconds for the application.
Note
The Qualification tool won’t catch all UDFs, and some of the UDFs can be handled with additional steps. Refer to Supported operators guide for more details on UDF.
By default, the applications and queries are sorted in descending order by the following fields:
Recommendation
Estimated GPU Time Saved
End Time
Stages Report#
For each stage used in SQL operations, the Qualification tool generates the following information. This file is saved per-application as qual_core_output/qual_metrics/<app-id>/stages.csv.
Stage ID
Stage Task Duration: amount of time spent in tasks of SQL Dataframe operations for the given stage.
Unsupported Task Duration: sum of task durations for the unsupported operators. For more details, see the Supported operators guide.
Stage Estimated: True or False indicates if we had to estimate the stage duration.
Execs Report#
The Qualification tool generates a report of the “Exec” in the “SparkPlan” or “Executor Nodes” along with the estimated acceleration on the GPU. Refer to the Supported operators guide for more details on limitations on UDFs and unsupported operators. This file is saved per-application as qual_core_output/qual_metrics/<app-id>/execs.csv.
SQL ID
Exec Name: example
Filter,HashAggregateExpression Name: for non-OSS Spark engines (Auron, Photon), this field can show the original operator name.
Exec Duration: wall-clock time measured since the operator starts until it is completed.
SQL Node Id
Exec Is Supported: whether the Exec is supported by RAPIDS or not. Refer to the Supported operators guide.
Exec Stages: an array of stage IDs.
Exec Children
Exec Children Node Ids
Exec Should Remove: whether the Op is removed from the migrated plan.
Exec Should Ignore: whether the Op is ignored from the migrated plan.
Action: action based on the Exec, can be
NONEif the Exec is supported,IgnoreNoPerfif the Exec is removed,IgnorePerfif the Exec is ignored, orTriageotherwise.
Operators Stats Report#
For each operator seen in a SQL query, the Qualification tool emits a per-application rollup saved as qual_core_output/qual_metrics/<app-id>/operators_stats.csv. Rows are grouped by the supported/unsupported status so the same operator name may appear twice when both supported and unsupported occurrences exist.
SQL ID: the SQL ID containing the operator.
Operator Type:
expressionorexec.Operator Name: the operator name (for example
Filter,Scan parquet,cast).Count: number of times the operator appears in the given SQL, scoped by its supported/unsupported status.
Supported: whether the operator is supported on GPU (
true/false).Stages: colon-separated list of stage IDs the operator participates in.
Statistics Report#
The CLI wrapper aggregates per-application, per-SQL, and per-operator statistics (derived from the per-app Execs, Stages, and Unsupported Operators files) into a single top-level qualification_statistics.csv alongside the run’s qualification_summary.csv.
App ID
SQL ID
Operator: operator name, for example
Filter,Scan parquet.Count: number of times the operator was executed in the given SQL ID.
Stage Task Exec Duration(s): total task time of all stages that contain the operator.
Total SQL Task Duration(s): total task time of the SQL that the operator is part of.
% of Total SQL Task Duration: percentage of stage task exec duration to total SQL task duration.
Supported: whether the operator is supported on the RAPIDS Accelerator for Spark plugin.
App Meta Data Report#
The qualification tool generates meta data for each processed app. This file is saved as app_metadata.json.
App ID
App Name
Eventlog: The event log file which contains the application.
Platform: The platform where the application was run.
Source Cluster Info: Cluster information of the application, including driver and worker node types, and the number of workers. This is either inferred by the Qualification tool from the event log or provided by the user through the tool command.
Recommended Cluster Info: Cluster information of the recommended GPU cluster by the Qualification tool.
Estimated GPU Speed-up Category: Large, Medium, or Small indicates the job should be migrated to GPU, or Not Recommended or Not Applicable otherwise.
Full Cluster Config Recommendations: This points to the file path of the cluster configs.
Sample output in text:
1[{
2 "appId": "application_000000000000_0001",
3 "appName": "sanity_test",
4 "eventLog": "file:/application_000000000000_0001",
5 "clusterInfo": {
6 "platform": "emr",
7 "sourceCluster": {
8 "driverNodeType": "i3.2xlarge",
9 "workerNodeType": "m5d.16xlarge",
10 "numWorkerNodes": 2
11 },
12 "recommendedCluster": {
13 "driverNodeType": "i3.2xlarge",
14 "workerNodeType": "g5.8xlarge",
15 "numWorkerNodes": 4
16 }
17 },
18 "estimatedGpuSpeedupCategory": "Not Recommended",
19 "fullClusterConfigRecommendations": "/qual_20240814145334_d2CaFA34/qual_core_output/tuning_apps/application_000000000000_0001/combined.conf",
20 "gpuConfigRecommendationBreakdown": "/qual_20240814145334_d2CaFA34/qual_core_output/tuning_apps/application_000000000000_0001/recommendations.log"
21}]
Parsing Expressions within each Exec
The Qualification tool looks at the expressions in each Exec to provide a fine-grained assessment of RAPIDS’ support.
It isn’t possible to extract the expressions for each available Exec: - some Execs don’t take any expressions, and - some execs may not show the expressions in the eventlog.
The following table lists the Execs’ name and the status of parsing their expressions where:
“Expressions Unavailable” marks the Execs that don’t show expressions in the eventlog
“Fully Parsed” marks the Execs that have their expressions fully parsed by the Qualification tool
“In Progress” marks the Execs that are still being investigated; therefore, a set of the marked Execs may be fully parsed in future releases.
Exec |
Expressions Unavailable |
Fully Parsed |
In Progress |
|---|---|---|---|
AggregateInPandasExec |
☑️ |
||
AQEShuffleReadExec |
☑️ |
||
ArrowEvalPythonExec |
☑️ |
||
BatchScanExec |
☑️ |
||
BroadcastExchangeExec |
☑️ |
||
BroadcastHashJoinExec |
☑️ |
||
BroadcastNestedLoopJoinExec |
☑️ |
||
CartesianProductExec |
☑️ |
||
CoalesceExec |
☑️ |
||
CollectLimitExec |
☑️ |
||
CreateDataSourceTableAsSelectCommand |
☑️ |
||
CustomShuffleReaderExec |
☑️ |
||
DataWritingCommandExec |
☑️ |
||
ExpandExec |
☑️ |
||
FileSourceScanExec |
☑️ |
||
FilterExec |
☑️ |
||
FlatMapGroupsInPandasExec |
☑️ |
||
GenerateExec |
☑️ |
||
GlobalLimitExec |
☑️ |
||
HashAggregateExec |
☑️ |
||
InMemoryTableScanExec |
☑️ |
||
InsertIntoHadoopFsRelationCommand |
☑️ |
||
LocalLimitExec |
☑️ |
||
MapInPandasExec |
☑️ |
||
ObjectHashAggregateExec |
☑️ |
||
ProjectExec |
☑️ |
||
PythonMapInArrowExec |
☑️ |
||
RangeExec |
☑️ |
||
SampleExec |
☑️ |
||
ShuffledHashJoinExec |
☑️ |
||
ShuffleExchangeExec |
☑️ |
||
SortAggregateExec |
☑️ |
||
SortExec |
☑️ |
||
SortMergeJoinExec |
☑️ |
||
SubqueryBroadcastExec |
☑️ |
||
SubqueryExec |
☑️ |
||
TakeOrderedAndProjectExec |
☑️ |
||
UnionExec |
☑️ |
||
WholeStageCodegenExec |
☑️ |
||
WindowExec |
☑️ |
||
WindowGroupLimitExec |
☑️ |
||
WindowInPandasExec |
☑️ |
||
WriteFilesExec |
☑️ |
UDF Report#
The Qualification tool generates a per-application UDF detection report saved as udf_report.json. The report captures UDFs discovered in the event log and the estimated time spent in stages that contain them.
udfs: list of detected UDFs. Each entry contains:
name: the UDF or expression name.
exec: the exec node hosting the UDF (for example,
Project,ArrowEvalPython).sql_id: the SQL query id.
stage_id: the stage id, or
nullwhen the stage can’t be resolved.
The list is empty when no UDFs are detected.
metrics: aggregate timing for stages containing UDFs, or
nullwhen no UDFs are detected. When present, containsunsupported_task_duration_ms,app_task_duration_ms, andunsupported_task_duration_pct. These estimates may include other unsupported ops present in the same stage.
MLFunctions Report#
The Qualification tool generates a report if there are SparkML or Spark XGBoost functions used in the eventlog. The functions in “spark.ml.” or “spark.XGBoost.” packages are displayed in the report.
App ID
Stage ID
ML Functions: List of ML functions used in the corresponding stage.
Stage Task Duration: amount of time spent in tasks containing ML functions for the given stage.
MLFunctions Total Duration Report#
The Qualification tool generates a report of total duration across all ML functions in the eventlog.
App ID
_Stage_Ids : Stage Id’s corresponding to the given ML function.
ML Function Name: ML function name supported on GPU.
Total Duration: total duration across all stages for the corresponding ML function.
Output Format#
The Qualification tool generates the output as CSV/log files.
Text and CSV files#
The Qualification tool generates a set of log/CSV files under the run directory. Wrapper-level summaries live at the top level; the core jar writes its files under the qual_core_output/ subdirectory, with per-application metrics under qual_core_output/qual_metrics/<app-id>/ and per-application tuning under qual_core_output/tuning_apps/<app-id>/. The content of each file is summarized in the following sections.
Application Report Summary#
The CLI wrapper prints a brief per-application summary to STDOUT and writes the same information to the top-level qualification_summary.csv at the root of the run directory.
The summary report outputs the following information: “App Name,” “App ID,” “App Duration,” “SQL DF duration,” “GPU Opportunity,” “Estimated GPU Duration,” “Estimated GPU Speedup,” “Estimated GPU Time Saved,” “Recommendation,” “Unsupported Execs,” “Unsupported Expressions,” “Estimated Job Frequency (monthly),” and “Total Core Seconds.”
1+------------+--------------+----------+----------+-------------+-----------+-----------+-----------+--------------------+-------------------------------------------------------+-----------+---------+
2| App Name | App ID | App | SQL DF | GPU | Estimated | Estimated | Estimated | Recommendation | Unsupported Execs |Unsupported Expressions| Estimated | Total |
3| | | Duration | Duration | Opportunity | GPU | GPU | GPU | | | | Job | Core |
4| | | | | | Duration | Speedup | Time | | | | Frequency | Seconds |
5| | | | | | | | Saved | | | | (monthly) | |
6+============+==============+==========+==========+=============+===========+===========+===========+====================+=======================================================+===========+=========+
7| appName-01 | app-ID-01-01 | 898429| 879422| 879422| 273911.92| 3.27| 624517.06|Strongly Recommended| | | 1 | 1843 |
8+------------+--------------+----------+----------+-------------+-----------+-----------+-----------+--------------------+-------------------------------------------------------+-----------+---------+
9| appName-02 | app-ID-02-01 | 9684| 1353| 1353| 8890.09| 1.08| 793.9| Not Recommended|Filter;SerializeFromObject;S...| hex | 30 | 9384 |
10+------------+--------------+----------+----------+-------------+-----------+-----------+-----------+--------------------+-------------------------------------------------------+-----------+---------+
In the above example, two application event logs were analyzed. “app-ID-01-01” is “Strongly Recommended” because Estimated GPU Speedup is ~3.27. On the other hand, the estimated acceleration running “app-ID-02-01” on the GPU isn’t high enough; hence the app isn’t recommended.
Per SQL Query Report Summary#
The Qualification tool has an option to generate a report at the per-SQL-query level. A brief summary of projected per-query performance on GPU is printed to STDOUT, and the detailed per-SQL CSV is written per-application as qual_core_output/qual_metrics/<app-id>/persql.csv.
The summary report outputs the following information: “App Name”, “App ID”, “Root SQL ID”, “SQL ID”, “SQL Description”, “SQL DF Duration”, “GPU Opportunity”, “Estimated GPU Duration”, “Estimated GPU Speedup”, “Estimated GPU Time Saved”, and “Recommendation”.
1+------------+--------------+----------+----------+---------------+----------+-------------+-----------+-----------+-----------+--------------------+
2| App Name | App ID | Root | SQL ID | SQL | SQL DF | GPU | Estimated | Estimated | Estimated | Recommendation |
3| | | SQL ID | | Description | Duration | Opportunity | GPU | GPU | GPU | |
4| | | | | | | | Duration | Speedup | Time | |
5| | | | | | | | | | Saved | |
6+============+==============+==========+==========+===============+==========+=============+===========+===========+===========+====================+
7| appName-01 | app-ID-01-01 | | 1| query41| 571| 571| 187.21| 3.05| 383.78|Strongly Recommended|
8+------------+--------------+----------+----------+---------------+----------+-------------+-----------+-----------+-----------+--------------------+
9| appName-02 | app-ID-02-01 | 3| 3| query44| 1116| 0| 1115.98| 1.0| 0.01| Not Recommended|
10+------------+--------------+----------+----------+---------------+----------+-------------+-----------+-----------+-----------+--------------------+
CSV Reports#
Entire App Report#
The per-application core summary is saved as qual_core_output/apps_summary.csv; the top-level qualification_summary.csv adds the QualX-derived speedup fields and ranks applications by Estimated GPU Speedup. Durations are reported in milliseconds.
Sample output in text can be referenced under: Application Report Summary
Per SQL Report#
The file is saved per-application as qual_core_output/qual_metrics/<app-id>/persql.csv.
Sample output in text can be referenced under: Per SQL Query Report Summary
Stages Report#
The file is saved per-application as qual_core_output/qual_metrics/<app-id>/stages.csv. Columns: Stage ID, Stage Task Duration, Unsupported Task Duration, Stage Estimated. See the Stages Report schema above for column descriptions.
Execs Report#
The file is saved per-application as qual_core_output/qual_metrics/<app-id>/execs.csv. Similar to the app and stage information, the table shows the estimated GPU performance of the SQL Dataframe operations. See the Execs Report schema above for the column list.
Status Report#
The file is saved globally as qual_core_output/status.csv. This contains file processing information for each eventlog.
1+-------------------------+-----------+----------------------------------------------------------------------------------+
2| Event Log | Status | Description |
3| | | |
4+=========================+===========+==================================================================================+
5| file:/home/app-ID-01-01 | "SUCCESS" | "app-ID-01-01, Took 500 ms to process" |
6+-------------------------+-----------+----------------------------------------------------------------------------------+
7| file:/home/app-ID-01-02 | "SKIPPED" | "GpuEventLogException: Cannot parse event logs from GPU run: skipping this file" |
8+-------------------------+-----------+----------------------------------------------------------------------------------+
Cluster Information Report#
The file is saved per-application as qual_core_output/qual_metrics/<app-id>/cluster_information.json. It captures driver/executor instance types, core and memory sizing, and (when available) the cluster ID and name.
ML Functions Report#
The file is saved per-application as qual_core_output/qual_metrics/<app-id>/ml_functions.csv. This contains information on ML functions detected in the eventlog.
ML Functions Duration Report#
The file is saved per-application as qual_core_output/qual_metrics/<app-id>/ml_functions_durations.csv. This contains total durations of all ML functions in the eventlog.
Unsupported Operators Report#
The file is saved per-application as qual_core_output/qual_metrics/<app-id>/unsupported_operators.csv. Each row records a single unsupported operator detected in the application’s execution plan, with the SQL ID, stage ID, operator type (Exec or Expression), operator name, a details string explaining why it was flagged, the stage task duration it contributed to, and the action (IgnoreNoPerf, IgnorePerf, or Triage).