Spark Connect Routing#

The Active Router path places the Smart Router, a gRPC proxy, between Spark Connect clients and CPU/GPU Spark Connect servers. The proxy requests a routing decision from the cuDF Router REST service and applies the recommended Spark configuration to the session. It can also record a CPU session and replay it on GPU after the primary run finishes.

Complete Installation and Configuration first. This integration targets Spark 4.0+ environments and requires compatible Spark Connect protocols across clients, the proxy, and upstream servers.

Configure the Upstream Servers#

Choose how the Router reaches each CPU and GPU endpoint:

Setting

Behavior

sparkConnectUri

Connect to an existing Spark Connect server.

clusterId

Use an existing platform cluster and let the Router manage its Spark Connect server lifecycle.

configFile

Let the Router provision a cluster and start Spark Connect from a platform-specific cluster configuration file.

On-premises deployments use existing clusters. EMR and Dataproc can use existing clusters or clusters created by the Router. Choose one endpoint setting per CPU/GPU entry and verify that your platform supports that mode.

Configure the Proxy#

Copy com.nvidia.grpc-proxy-*.jar and aether-connect-*.jar from the installed wheel’s aether/resources/smart-router/ directory to a runtime directory. Configure a TLS certificate and key for the proxy’s hostname.

The following smart-router.yaml connects to existing Spark Connect servers. Replace the paths, hosts, and ports for your environment:

proxy:
  port: 15002
  certChainPath: /path/to/server.crt
  privateKeyPath: /path/to/server.key

aether:
  apiUrl: http://localhost:8000

cluster:
  cpu:
    sparkConnectUri: sc://cpu-spark-host:15002
  gpu:
    sparkConnectUri: sc://gpu-spark-host:15002

shadowRun:
  enabled: false

Start the REST service, then run the proxy with Java 17 or newer, supplying the paths to both JARs:

java -Dconfig.file=/path/to/smart-router.yaml \
  -cp '/path/to/com.nvidia.grpc-proxy-<version>.jar:/path/to/aether-connect-<version>.jar' \
  com.nvidia.smartrouter.GrpcGateway

Clients must reach the proxy’s gRPC port. The proxy must reach the REST service and both Spark Connect servers. The Router must be able to read the event logs generated by both execution environments.

Connect a Workload#

Point your Spark Connect client at the proxy. Supply the appropriate token for your deployment and use the same connect_id for repeated runs of a logical job:

export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/path/to/trusted-ca-or-server.crt
export SPARK_REMOTE='sc://<router-host>:15002/;use_ssl=true;token=<token>;connect_id=<job-id>'
spark-submit your_job.py

End the session with spark.stop() so that completion processing can release the primary endpoint and process the run. A new job normally runs on CPU while the Router gathers evidence. Later executions can move to GPU after evaluation and validation satisfy the routing policy.

Enable GPU Shadow Runs#

A shadow run replays the captured session on GPU to evaluate it without changing the primary run’s CPU route. Before enabling shadow runs, configure a supported GPU replay environment and its isolation policy.

For workloads that write data, configure the platform’s sandbox artifacts and platform.<platform>.cluster.sandbox.warehouse_path. Then enable sandboxed replay:

shadowRun:
  enabled: true
  requiresSandbox: true

Sandbox replay uses staging and workload phases with sandbox-aware GPU Spark Connect servers. It redirects supported table operations to sandbox storage. Check the workload’s operations and platform support before using this mode; it does not make arbitrary external side effects safe to replay.

Inspect the GPU profile, validation result, and migration readiness as described in Operations and Reporting. Enabling shadow runs alone does not guarantee a GPU route.