Installation and Configuration#
Prerequisites#
Prepare the following before configuring the Router:
A Python environment supported by the downloaded bundle. The current release includes offline dependencies for Python 3.10 through 3.12.
A supported Spark environment and GPU nodes with the cuDF plugin and its runtime dependencies installed. See Overview for the Spark plugin requirements.
Credentials and network access for your platform, event-log storage, output storage, and any cluster or job lifecycle operations you enable.
CPU Spark event logs for initial qualification, and GPU execution capacity for profiling, tuning, and validation.
Spark compatibility depends on the integration:
Component |
Requirement |
|---|---|
Active Spark Connect routing |
Spark 4.0+ with compatible client, proxy, and upstream Spark Connect protocols; Java 17+ for the Smart Router. |
Capture and replay |
Spark 3.5+ capture hosts and compatible Spark Connect replay targets. |
Sandbox replay |
Iceberg sandboxing on Spark 3.5+; Delta sandboxing on Spark 4.0+ with Delta on the classpath. Platform support must also be verified. |
These requirements are separate from the Spark plugin’s general compatibility matrix. Support for event-log analysis alone does not imply support for cluster provisioning, Spark Connect lifecycle, or sandbox replay.
Install the Resource Bundle#
Download and extract the cuDF Router resource
from NGC. Use the install script and wheel supplied in the same bundle.
Replace <version> and <platform> in the example with your release and
platform, such as onprem, dataproc, or emr:
bash cudf-router_v<version>/install-<version>.sh \
-w cudf-router_v<version>/nvidia_aether-<version>-py3-none-any.whl \
-p <platform>
source venv-aether/bin/activate
aether version
aether config init
aether config set core.selected_platform <platform>
The resource bundle contains the Router service and CLI, dependencies, and runtime JARs for routing, sandboxing, and workload capture and replay. Follow the release-specific installation instructions for the exact artifact names and available deployment options.
Configure Storage and Job History#
Enable Spark event logging for your workloads and select storage readable by the Router. Configure the event-log directory for the selected platform:
aether config set platform.<platform>.spark_event_log_dir <event-log-directory-uri>
aether config list
The Router also needs write access to its output and cache locations and to platform artifact locations used for submission or replay. Configure platform credentials and storage before attempting GPU execution.
SQLite is the default database for local single-user use. For a shared REST service or routing deployment, use PostgreSQL. The REST service and workers must share the same durable database and compatible configuration:
aether config set database.job_history_db_type postgresql
aether config set database.job_history_connection <postgres-connection>
After selecting and configuring the database, initialize its schema:
aether migrate apply
Start the REST Service#
For local development:
aether config set rest.enable_docs true
aether rest run
The interactive API reference is available at
http://localhost:8000/docs when enabled. Use aether rest run --help
to configure the listen address for your deployment.
See Operations and Reporting for service checks.
Next Steps#
Continue with Spark Connect Routing for transparent Spark Connect routing, or API Integration with Capture and Replay to integrate your own submission platform.