> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Schedule Prediction Jobs

> Configure recurring batch predictions with your preferred scheduler

Schedule the tested script or notebook from [Production Workflow](/kumo-engine/1.0.0/user-code-guide/deploy-a-model/overview) after its upstream data refresh completes. Kumo prediction runs synchronously; your scheduler launches the workflow and tracks completion.

## Define the Job Inputs

Keep these settings explicit in the job configuration:

| Setting | Specify                                                                                               |
| ------- | ----------------------------------------------------------------------------------------------------- |
| Model   | Exact bundle path or registry version.                                                                |
| Data    | Source locations, compatible graph and query definitions, and upstream readiness condition.           |
| Scoring | Population filter, task options, and temporal anchor where applicable.                                |
| Output  | Logical batch identifier, attempt destination, and delivery policy.                                   |
| Runtime | Packages, compute, local SSD scratch, required embedding models, execution identity, and permissions. |

For temporal queries, `prediction_time` is a timezone-naive Python `datetime` interpreted as UTC. Keep it separate from the scheduler's start time. Static queries do not take a prediction time. See [Select the Population](/kumo-engine/1.0.0/user-code-guide/making-predictions#select-the-population).

## Configure the Scheduler

Use your preferred scheduler or orchestrator. The options below are examples; any scheduler that can run or submit the workflow and track completion is suitable.

Set the schedule and time zone, upstream dependencies, timeout, retry policy, and failure notifications. Limit overlapping runs when they share compute, scratch, or output destinations.

### Snowflake ML Jobs

Follow the [Snowflake production workflow](/kumo-engine/1.0.0/user-code-guide/deploy-a-model/overview#example-productionize-your-model-in-snowflake) to prepare and test the prediction job. Register it as an ML Job Definition, then invoke that definition from a scheduled Snowflake Task or task graph using [Snowflake's integration guidance](https://docs.snowflake.com/en/developer-guide/snowflake-ml/ml-jobs/overview#ml-job-definitions).

Supply the intended batch's `--prediction-time` and a distinct `--output-table` on each scheduled invocation; do not reuse the example's fixed date and table for every run.

Set the task's schedule after the source refresh, or use a task dependency when the refresh is part of the same task graph. Confirm that the scheduled execution role can read the sources and model registry and write the output destination. Run the task once and check the completed ML Job and saved batch before enabling recurring delivery.

### Databricks Scheduled Notebook

Create a notebook task in a Databricks job and [add a scheduled trigger](https://docs.databricks.com/aws/en/jobs/scheduled). Configure the runtime and execution identity using [Databricks Notebook Setup](/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/databricks-notebook).

The notebook loads the retained model from its Unity Catalog volume path, rebuilds the snapshot, and writes predictions using the job identity's permissions.

### Apache Airflow

Define a [DAG](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html) with dependencies for source refresh, prediction, output validation, and delivery. Run the prediction script in a prepared Engine environment or submit it to your execution platform.

For remote jobs, wait for completion and propagate failures before validating outputs. The execution environment needs the packages, compute, scratch, and credentials; successful submission alone does not establish job success.

## Validate Outputs and Handle Retries

Assign one logical batch identifier to each scoring request, such as a daily customer scoring batch. Record its model version, source or snapshot version, population settings, and temporal anchor where applicable. Keep that identifier across retries, and give each attempt its own output path or table so an incomplete attempt cannot replace an accepted result. These identifiers and acceptance records belong to your job configuration and delivery system.

Read each saved batch through the application's access path. Check the expected population, keyed scores, and task-specific output counts. Record the attempt destination, row counts, duration, and status. Alert on failures and missed freshness deadlines.

Choose output behavior before enabling retries:

* **Files and object storage:** use a new attempt path and retain every output part. Follow the [file layout and ownership rules](/kumo-engine/1.0.0/user-code-guide/advanced-prediction-configurations#choose-file-formats-and-batch-paths).
* **Tables:** prefer a separate attempt table until validation succeeds. If the application requires append or replacement of a shared table, account for partial writes and concurrent writers before retrying. See [export behavior](/kumo-engine/1.0.0/user-code-guide/advanced-prediction-configurations#export-an-existing-result).
* **Delivery:** update the application's accepted batch location only after validation, and prevent a second attempt from delivering the same logical batch again. Keep the last accepted batch or use the application's agreed fallback on failure.

For a retry of the same scoring request, keep the recorded model, population settings, temporal anchor, and source version or retained snapshot. Rebuilding a snapshot from changed source data changes the scoring inputs. If scoring completed and the local result files remain available, [export the existing result](/kumo-engine/1.0.0/user-code-guide/advanced-prediction-configurations#export-an-existing-result) to a new attempt destination without scoring again.

Keep this prediction schedule separate from [model refresh and acceptance](/kumo-engine/1.0.0/user-code-guide/deploy-a-model/refresh-a-model#evaluate-and-replace-the-model). A new data batch does not by itself require a new model.