Overview#

cuDF Router helps you evaluate Spark workloads for GPU acceleration and route recurring jobs to CPU or GPU. It brings event-log analysis, configuration recommendations, GPU evaluation, and run history into one workflow for the cuDF plugin for Apache Spark.

Qualification estimates GPU suitability from CPU event logs. The Router then uses evidence from GPU runs, including profiling and validation, to decide where subsequent runs should execute. A qualification estimate is a starting point; measure GPU performance and validate results before migrating a job.

Choose an Integration#

Integration

Choose this when

How it works

Spark Connect routing

Your Spark clients can connect through a proxy.

The Smart Router proxies sessions to CPU or GPU Spark Connect servers.

API integration

Your platform or scheduler controls job submission.

Your platform requests a route before each run and reports completion afterward. Capture and replay can supply GPU evaluation runs.

Both paths use the same REST service and job-history database. The Router needs access to event logs and a GPU execution environment with the cuDF plugin installed. See Installation and Configuration for prerequisites and configuration.

From CPU Evaluation to GPU Routing#

  1. Identify the job. Use a stable logical job identifier across repeated executions, and a distinct run identifier for each execution.

  2. Collect a CPU baseline. Analyze its event log to estimate GPU potential and generate an initial configuration.

  3. Evaluate on GPU. Run a GPU trial, or replay a captured CPU workload as a shadow run. Configure an isolated sandbox for supported workloads that write data.

  4. Profile and validate. Analyze GPU performance, compare CPU/GPU results or event-log metrics, and refine the configuration.

  5. Route later executions. The Router combines the recorded evidence and routing policy to choose CPU or GPU and return the configuration to use.

Use the operations guide to inspect readiness, review reports, and diagnose jobs that remain on CPU.

Existing Tools Users#

cuDF Router is the entry point for the replacement workflow. The following standalone tool references are retained in Appendixes for existing users:

The Router has its own commands, configuration, and persistent job history. Use its installation instructions rather than the archived tools’ setup instructions.

Downloads#

Get the released bundle from the cuDF Router resource on NGC. The examples in this section use the aether command shipped in the 26.08.0 resource bundle. Check the resource’s release instructions when using a different version.