Benchmark Environments#

Cluster information for the ETL benchmarks used for the qualification estimation model. Note that all benchmarks were run using the NDS benchmark at SF3K (3 TB scale).#

Environment

CPU Cluster

GPU Cluster

On-prem

8x 128-core

8x 128-core + 8x A100 40 GB

Dataproc (T4)

4x n1-standard-32

4x n1-standard-32 + 8x T4 16GB

Dataproc (L4)

8x n1-standard-16

8x g2-standard-16

EMR

8x m5d.8xlarge

4x g4dn.12xlarge

Databricks AWS

8x m6gd.8xlage

8x g5.8xlarge

Databricks Azure

8x E8ds_v4

8x NC8as_T4_v3