API Integration with Capture and Replay#
Use API integration when a scheduler or platform owns Spark submission. The platform calls the cuDF Router before and after each execution; it does not need to put a proxy in the live client path. The Router owns the routing decision and evidence processing, while the platform owns submission, execution controls, and isolation.
Complete Installation and Configuration and start the REST service before integrating the lifecycle calls. Use the interactive API reference from your installed version for complete request and response schemas.
Identify Jobs and Runs#
Keep aether_job_id stable for repeated executions of the same logical
job. Set custom_job_run_id to a distinct value for each execution, and
use that same pair when requesting a route and reporting completion. This
lets the Router associate GPU evidence and recommendations with the correct
job history.
Request a Route Before Submission#
Call GET /api/v2/route_job before submitting a job:
curl -sS \
'http://<router-api-host>:8000/api/v2/route_job?aether_job_id=<job-id>&custom_job_run_id=<run-id>' \
| python3 -m json.tool
Use the returned runtime and Spark configuration when constructing the job:
Response |
Platform action |
|---|---|
|
Run on CPU and merge |
|
Run on CPU. Apply any returned capture properties. |
|
Run on GPU with the recommended configuration and the cuDF plugin installed. |
HTTP |
If capture-on-unknown is disabled, use the normal CPU path and report completion so the Router can collect evidence. |
Handle service failures according to your platform’s availability policy; an unavailable routing service is different from a response indicating an unknown job.
Report Completion#
After execution, call POST /api/v2/process_completed_job with the job/run
identifiers, actual completion status, and event-log location or supported
platform job identifier. For a successful run with shadow evaluation enabled:
curl -sS -X POST \
'http://<router-api-host>:8000/api/v2/process_completed_job' \
-H 'Content-Type: application/json' \
-d '{
"aether_job_id": "<job-id>",
"custom_job_run_id": "<run-id>",
"job_status": "finished",
"event_log": "<event-log-uri>",
"enable_shadow_run": true
}'
Report failed runs too, using the failure fields supported by your installed API. Do not mark a failed execution as finished successfully.
Completion processing is asynchronous. Poll its status before treating qualification, profiling, or shadow evaluation as complete:
curl -sS \
'http://<router-api-host>:8000/api/v2/process_completed_job/status?aether_job_id=<job-id>' \
| python3 -m json.tool
Enable Capture and GPU Replay#
Spark Workload Replay captures a CPU workload for later execution against a GPU Spark Connect server. Configure capture storage and the capture-plugin JAR location for your platform:
aether config set platform.<platform>.provide_capture_plugin true
aether config set platform.<platform>.set_capture_on_unknown true
aether config set platform.<platform>.spark_workload_capture_dir <capture-output-uri>
aether config set platform.<platform>.workload_replay_capture_jar_uri <capture-plugin-jar-uri>
aether config set migration.min_shadow_runs 1
aether config set migration.max_shadow_runs 2
The platform must apply the capture properties returned by route_job
and make the capture plugin available to Spark. Only omit the JAR URI when
your platform already installs or stages the plugin through another mechanism.
For Router-managed replay, provide cluster registration or provisioning and a Spark Connect server lifecycle. The relevant API operations are:
POST /api/v2/clusterorPOST /api/v2/cluster/register.POST /api/v2/spark_connect_server.POST /api/v2/spark_connect_server/spark_workload_replay.DELETE /api/v2/spark_connect_server.DELETE /api/v2/clusterfor clusters created by the Router.
Replay can write data. Configure a supported isolated GPU sandbox with a writable sandbox warehouse, appropriate production read access, and platform controls for credentials, quotas, and lifecycle cleanup before enabling replay for writing workloads.
After replay, the Router profiles the GPU event log, validates the paired CPU/GPU runs, and updates routing evidence. Your platform can inspect the result through the readiness and reporting interfaces in Operations and Reporting.