Is this page helpful?

Qualification Tool — Supported Features#

Delta Lake Support#

This section outlines how the Qualification tool handles Delta Lake features and operations.

Version Compatibility

The Qualification tool takes a flexible approach to version compatibility:

  • No version validation: The tool does not validate Spark or Delta Lake versions because GPU and CPU clusters often run different software stacks.

  • No configuration validation: Spark configurations are not validated since GPU applications typically require different settings than CPU applications.

Reading Operations#

Delta Metadata Scans

The Qualification tool identifies Delta Lake metadata operations by detecting specific keywords (_delta_log, DeltaLogFileIndex) in file paths. When detected, the tool:

  • Marks the entire query as a Delta metadata scan (unsupported)

  • Flags the operation as unsupported on GPU

Parquet File Scans

When Delta Lake tables read their underlying Parquet files, the Qualification tool handles them as follows:

  • Treats as supported: These operations appear as standard Parquet reads in event logs and are marked as GPU-supported

  • Current limitations:

    • Cannot identify tables with deletion vectors (assumes all reads will work on GPU)

    • Does not recognize compaction operations

Write Operations#

The Qualification tool marks most write operations as supported on GPU, but has several current limitations:

General Limitations

  • Missing event log details: Some write operators lack sufficient information in Spark event logs

  • Table properties: Cannot identify table properties and configurations (assumes all operations will work on GPU)

  • Compaction operations: Does not recognize or properly handle compaction

Supported Write Operations

The following table shows which Delta Lake write operations are currently supported:

Delta Lake Write Operations Support#

Operation

Supported

Detection Method

Future Improvements

AppendDataExecV1

☑️ Supported

Detects DeltaTableV2 pattern in plan info

OverwriteByExpressionExecV1

☑️ Supported

Detects DeltaTableV2 pattern in plan info

AtomicCreateTableAsSelect

☑️ Supported

Detects *delta* pattern in plan info

AtomicReplaceTableAsSelect

☑️ Supported

Detects *delta* pattern in plan info

SaveIntoDataSourceCommand

☑️ Supported

Basic detection (no specific pattern)

Parse physical plan to extract arguments

DeleteCommand

☑️ Supported

Basic detection (no specific pattern)

Parse physical plan to extract arguments

UpdateCommand

☑️ Supported

Basic detection (no specific pattern)

Parse physical plan to extract arguments

MergeIntoCommand

☑️ Supported

Basic detection (no specific pattern)

Parse physical plan to extract arguments

OptimizeTableCommand

☑️ Supported

Basic detection (no specific pattern)

Parse physical plan to extract arguments

DescribeDeltaHistoryCommand

❌ Not Supported

Not applicable

None planned

ShowPartitionsDeltaCommand

❌ Not Supported

Not applicable

None planned