API Integration with Capture and Replay#

Use API integration when a scheduler or platform owns Spark submission. The platform calls the cuDF Router before and after each execution; it does not need to put a proxy in the live client path. The Router owns the routing decision and evidence processing, while the platform owns submission, execution controls, and isolation.

Complete Installation and Configuration and start the REST service before integrating the lifecycle calls. Use the interactive API reference from your installed version for complete request and response schemas.

Identify Jobs and Runs#

Keep aether_job_id stable for repeated executions of the same logical job. Set custom_job_run_id to a distinct value for each execution, and use that same pair when requesting a route and reporting completion. This lets the Router associate GPU evidence and recommendations with the correct job history.

Request a Route Before Submission#

Call GET /api/v2/route_job before submitting a job:

curl -sS \
  'http://<router-api-host>:8000/api/v2/route_job?aether_job_id=<job-id>&custom_job_run_id=<run-id>' \
  | python3 -m json.tool

Use the returned runtime and Spark configuration when constructing the job:

Response

Platform action

next_run_type=CAPTURE

Run on CPU and merge recommended_spark_configuration into the job configuration to enable workload capture.

next_spark_runtime=SPARK

Run on CPU. Apply any returned capture properties.

next_spark_runtime=SPARK_RAPIDS

Run on GPU with the recommended configuration and the cuDF plugin installed.

HTTP 404 for an unknown job

If capture-on-unknown is disabled, use the normal CPU path and report completion so the Router can collect evidence.

Handle service failures according to your platform’s availability policy; an unavailable routing service is different from a response indicating an unknown job.

Report Completion#

After execution, call POST /api/v2/process_completed_job with the job/run identifiers, actual completion status, and event-log location or supported platform job identifier. For a successful run with shadow evaluation enabled:

curl -sS -X POST \
  'http://<router-api-host>:8000/api/v2/process_completed_job' \
  -H 'Content-Type: application/json' \
  -d '{
    "aether_job_id": "<job-id>",
    "custom_job_run_id": "<run-id>",
    "job_status": "finished",
    "event_log": "<event-log-uri>",
    "enable_shadow_run": true
  }'

Report failed runs too, using the failure fields supported by your installed API. Do not mark a failed execution as finished successfully.

Completion processing is asynchronous. Poll its status before treating qualification, profiling, or shadow evaluation as complete:

curl -sS \
  'http://<router-api-host>:8000/api/v2/process_completed_job/status?aether_job_id=<job-id>' \
  | python3 -m json.tool

Enable Capture and GPU Replay#

Spark Workload Replay captures a CPU workload for later execution against a GPU Spark Connect server. Configure capture storage and the capture-plugin JAR location for your platform:

aether config set platform.<platform>.provide_capture_plugin true
aether config set platform.<platform>.set_capture_on_unknown true
aether config set platform.<platform>.spark_workload_capture_dir <capture-output-uri>
aether config set platform.<platform>.workload_replay_capture_jar_uri <capture-plugin-jar-uri>
aether config set migration.min_shadow_runs 1
aether config set migration.max_shadow_runs 2

The platform must apply the capture properties returned by route_job and make the capture plugin available to Spark. Only omit the JAR URI when your platform already installs or stages the plugin through another mechanism.

For Router-managed replay, provide cluster registration or provisioning and a Spark Connect server lifecycle. The relevant API operations are:

  • POST /api/v2/cluster or POST /api/v2/cluster/register.

  • POST /api/v2/spark_connect_server.

  • POST /api/v2/spark_connect_server/spark_workload_replay.

  • DELETE /api/v2/spark_connect_server.

  • DELETE /api/v2/cluster for clusters created by the Router.

Replay can write data. Configure a supported isolated GPU sandbox with a writable sandbox warehouse, appropriate production read access, and platform controls for credentials, quotas, and lifecycle cleanup before enabling replay for writing workloads.

After replay, the Router profiles the GPU event log, validates the paired CPU/GPU runs, and updates routing evidence. Your platform can inspect the result through the readiness and reporting interfaces in Operations and Reporting.