Spark Connect Routing#
The Active Router path places the Smart Router, a gRPC proxy, between Spark Connect clients and CPU/GPU Spark Connect servers. The proxy requests a routing decision from the cuDF Router REST service and applies the recommended Spark configuration to the session. It can also record a CPU session and replay it on GPU after the primary run finishes.
Complete Installation and Configuration first. This integration targets Spark 4.0+ environments and requires compatible Spark Connect protocols across clients, the proxy, and upstream servers.
Configure the Upstream Servers#
Choose how the Router reaches each CPU and GPU endpoint:
Setting |
Behavior |
|---|---|
|
Connect to an existing Spark Connect server. |
|
Use an existing platform cluster and let the Router manage its Spark Connect server lifecycle. |
|
Let the Router provision a cluster and start Spark Connect from a platform-specific cluster configuration file. |
On-premises deployments use existing clusters. EMR and Dataproc can use existing clusters or clusters created by the Router. Choose one endpoint setting per CPU/GPU entry and verify that your platform supports that mode.
Configure the Proxy#
Copy com.nvidia.grpc-proxy-*.jar and aether-connect-*.jar from the
installed wheel’s aether/resources/smart-router/ directory to a runtime
directory. Configure a TLS certificate and key for the proxy’s hostname.
The following smart-router.yaml connects to existing Spark Connect
servers. Replace the paths, hosts, and ports for your environment:
proxy:
port: 15002
certChainPath: /path/to/server.crt
privateKeyPath: /path/to/server.key
aether:
apiUrl: http://localhost:8000
cluster:
cpu:
sparkConnectUri: sc://cpu-spark-host:15002
gpu:
sparkConnectUri: sc://gpu-spark-host:15002
shadowRun:
enabled: false
Start the REST service, then run the proxy with Java 17 or newer, supplying the paths to both JARs:
java -Dconfig.file=/path/to/smart-router.yaml \
-cp '/path/to/com.nvidia.grpc-proxy-<version>.jar:/path/to/aether-connect-<version>.jar' \
com.nvidia.smartrouter.GrpcGateway
Clients must reach the proxy’s gRPC port. The proxy must reach the REST service and both Spark Connect servers. The Router must be able to read the event logs generated by both execution environments.
Connect a Workload#
Point your Spark Connect client at the proxy. Supply the appropriate token
for your deployment and use the same connect_id for repeated runs of a
logical job:
export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/path/to/trusted-ca-or-server.crt
export SPARK_REMOTE='sc://<router-host>:15002/;use_ssl=true;token=<token>;connect_id=<job-id>'
spark-submit your_job.py
End the session with spark.stop() so that completion processing can
release the primary endpoint and process the run. A new job normally runs
on CPU while the Router gathers evidence. Later executions can move to GPU
after evaluation and validation satisfy the routing policy.
Enable GPU Shadow Runs#
A shadow run replays the captured session on GPU to evaluate it without changing the primary run’s CPU route. Before enabling shadow runs, configure a supported GPU replay environment and its isolation policy.
For workloads that write data, configure the platform’s sandbox artifacts
and platform.<platform>.cluster.sandbox.warehouse_path. Then enable
sandboxed replay:
shadowRun:
enabled: true
requiresSandbox: true
Sandbox replay uses staging and workload phases with sandbox-aware GPU Spark Connect servers. It redirects supported table operations to sandbox storage. Check the workload’s operations and platform support before using this mode; it does not make arbitrary external side effects safe to replay.
Inspect the GPU profile, validation result, and migration readiness as described in Operations and Reporting. Enabling shadow runs alone does not guarantee a GPU route.