Production on Ray and Anyscale

Run each Engine workload on one GPU worker in a GCP Ray cluster. Use Anyscale Jobs for managed scheduling and operations, or operate the Ray cluster yourself.

Prepare GCP

Adapt the following steps to your cloud configuration. Each Engine workload runs on one GPU worker; the Ray head coordinates execution.

Provision resources

1. Select a GCP project and region with sufficient GPU quota.
2. Provision a VPC, subnet and network routes for cluster communication, image access, data sources and tracking endpoints.
3. Create a worker service account and GCS buckets for retained models, artifacts and prediction outputs.
4. Select GPU worker machines with local SSD scratch. Keep Engine computation on workers rather than the head node.

Size compute and storage

1. Use GPU sizing to select CPU, RAM and local SSD capacity for one job. For 100 GiB of source Parquet, the guidance recommends 32 vCPUs, at least 150 GiB RAM and 400 GiB free scratch, increasing to 800 GiB for graphs with many edges.
2. Select GPU memory for your model and measure peak use with representative data. The source-data sizing estimates do not specify GPU memory requirements.
3. Place scratch on local SSD and retain models and accepted outputs in durable storage. Network storage and cloud block volumes are not supported as Engine working storage.
4. Size concurrent jobs for their combined resource use. Additional workers do not automatically distribute one Engine graph.

Configure access

1. Grant workers access to the required source data and GCS locations. Head-node permissions do not grant access to workers.
2. Configure service-account authentication for the job and access to the container registry.
3. Configure private connectivity and approved outbound access for packages, data and tracking. For Anyscale, include the control-plane connectivity described in GCP cloud setup.

Build the job image

1. Select a Ray image with a Python and CUDA stack compatible with the approved Engine build.
2. Install kumo-engine[gcs] for GCS access. Include the mlflow extra if using MLflow tracking, and other connector extras required by your sources. For example, kumo-engine[gcs,mlflow] includes both integrations.
3. Package the same Ray version used by the cluster. For self-managed installations, include Ray’s cluster/job tooling through ray[default]; retain the Ray installation supplied by an Anyscale base image.
4. Copy the prediction script and graph/query modules into the image. Record its version or digest and the installed dependency versions.
5. Run the Engine import and GPU checks from Linux setup inside the image.

Prepare the model and prediction script

1. Select an evaluated model and retain its complete bundle, exact version and required text embedding assets. Store these outside temporary scratch. See Publish and Retrieve Models.

2. Move the working prediction code into predict_batch.py, including imports, connector setup, graph definitions and predictive-query definitions. The production script illustrates the structure; adapt its connectors to your data sources.

3. Supply the following inputs through your script arguments or job configuration:

   Input	Value
   Model	Exact accepted registry version or bundle path
   Data	Source tables and recorded source version or retained snapshot
   Prediction time	Fixed UTC anchor for temporal queries; omitted for static queries
   Batch identifier	One identifier for the scoring request, retained across retries
   Attempt destination	A separate output table or path for each attempt
   Scratch	Writable node-local SSD directory

   These are application inputs, not built-in Engine CLI flags. A model bundle does not contain the source data or graph snapshot.

4. Load the model and build or reload the intended snapshot. Regenerate the prediction population for the batch.

5. Call Trainer.predict inside the Kumo MLOps context to retain prediction records for monitoring. Configure a durable artifact root and the tracking destination for your environment.

6. Export predictions to the attempt destination. Check the saved keys, population and task-specific row counts before making the output available to the application.

Submit an Engine workload to a worker

1. Create Engine objects within the worker process. Pass model references and artifact URIs from the driver rather than serializing a loaded trainer.

2. Add a Ray wrapper around the prediction script. The following wrapper accepts your application’s script arguments after the wrapper filename:

   import subprocess
   import sys
   import ray
   @ray.remote(num_gpus=1)
   def score(arguments):
       subprocess.run(
           [sys.executable, "predict_batch.py", *arguments], check=True
       )
   if __name__ == "__main__":
       ray.init(address="auto")
       ray.get(score.remote(sys.argv[1:]))

3. Set task CPU and memory requirements for the selected worker before enabling concurrency. The example reserves one GPU; local SSD capacity is provisioned on the worker.

4. Include the wrapper and prediction script in the same job working directory. Use the wrapper as the entrypoint for either deployment path below.

Deploy on Anyscale

1. Configure an Anyscale GCP cloud. Use Compute Engine for this workflow, or your existing GKE environment if that is your organization’s standard.
2. Define the job image, wrapper entrypoint, application arguments, GPU worker type and maximum worker count in the job configuration.
3. Submit the job and review terminal status, logs and saved outputs.
4. Configure a schedule, failure alerts and bounded retries after completing the checks below.
5. Monitor compute utilization, startup time, storage and transfer costs alongside the batch deadline.

Anyscale retries restart the entrypoint. Required state must be reloadable from durable storage, and output publication must tolerate repeated execution.

Deploy on self-managed Ray

1. Configure the GCP cluster launcher with GPU workers and local SSD storage. Replace the guide’s small CPU example with your selected worker configuration.
2. Configure worker service accounts and a bounded worker count. Set limits according to quota and the number of independent jobs.
3. Start the cluster and submit the wrapper through Ray Jobs.
4. Configure your scheduler to wait for terminal job status and propagate failures before accepting output.
5. Configure cluster startup/shutdown, log retention, alerts and recovery. Assign an operator for the cluster lifecycle.

Run the job before scheduling

1. Run the job from a fresh runtime with the intended execution identity.
2. Review terminal job status, saved predictions and tracking records. Read the saved output through the application’s access path.
3. Measure peak RAM, GPU memory, scratch use and job duration on representative data. Adjust capacity before increasing concurrency.
4. Test an interrupted run and retry before publishing its output. Keep the same model, data version, population and prediction anchor across attempts. Accept only one output for each logical batch.
5. Test rollback to the previous model version and accepted output.

Schedule predictions and monitor results

1. Schedule the job after the upstream data refresh, or configure a dependency on the refresh task.
2. Record the model version, data or snapshot version, prediction anchor, batch identifier, accepted output location and job status together.
3. Configure operational alerts for failed jobs and missed freshness deadlines. Track job duration, resource use and requested, scored and skipped rows.
4. Compute score drift from retained Kumo prediction records. Evaluate model performance when labels become available and the prediction outcome window has completed.
5. Export monitoring results to governed tables and configure their refresh schedule, alert thresholds and responsible operator.

Job logs describe execution health; drift and labeled outcomes describe model behavior. Investigate data freshness and population changes before deciding to retrain. Registry-native scoring may use a different monitoring path from jobs calling Trainer.predict inside Kumo MLOps.

For output delivery, keep each attempt separate until validation succeeds. Publish the accepted location once per batch and retain the previous accepted output on failure. See Schedule Prediction Jobs.

Promote and refresh models

1. Train and register a candidate without changing the production alias.
2. Compare the candidate with the current model using the same evaluation population and outcome definition.
3. Run the candidate in the production job environment, then promote the accepted version. Record the exact version used for each prediction batch.
4. Retain the previous model, embedding assets and runtime configuration. To roll back, select that version for subsequent jobs and restore the previous accepted output where needed.

See Refresh a Model for refresh cadence and evaluation guidance.

Assign ISV operating responsibilities

1. Assign the following responsibilities before deployment:

   Owner	Responsibility
   Customer administrator	Compute, storage, networking, identities and resource grants
   ISV deployer	Compatible runtime, job code and deployment configuration
   Production operator	Schedules, alerts, promotion, rollback, retention and support

2. For ISV-operated environments, separate customer credentials, artifact roots, outputs and execution resources. A shared Python process does not provide tenant isolation.

3. Store credentials in managed secrets or use workload identity. Define credential rotation and revocation procedures, and exclude tokens and customer rows from diagnostic logs.

4. Agree on compute ownership, artifact retention and deletion permissions.

Troubleshoot jobs