> This page is for Kumo Engine.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Production on Snowflake

> Integrate Engine model artifacts into production jobs, provisioning and observability

Run an existing Engine model as a GPU ML Job and schedule it with Snowflake Tasks. The query warehouse supplies data access; the compute pool runs Engine.

## Prepare Snowflake

Adapt the following steps to your account configuration. This guide uses GPU ML Jobs, Snowflake Tasks, Model Registry and Snowflake tracking.

### Provision resources

1. Choose a compatible GPU Container Runtime and Python build from [Snowflake setup](/sdgm/kumo-engine/get-started-with-kumo-engine/installation/snowflake-notebook).
2. Provision a GPU compute pool available in your region. Use the sizing steps below to select its resources.
3. Provision a query warehouse, source schemas, model registry schema, output schema and payload/artifact stages. The warehouse handles connector queries; the compute pool runs Engine.
4. Configure the stage used by the Engine artifact adapter with `SNOWFLAKE_SSE` encryption. Keep retained models and outputs separate from job-local scratch.

### Size compute and storage

1. Use [GPU sizing](/sdgm/kumo-engine/get-started-with-kumo-engine/gpu-sizing) to select CPU, RAM and local SSD capacity for one job. For 100 GiB of source Parquet, the guidance recommends 32 vCPUs, at least 150 GiB RAM and 400 GiB free scratch, increasing to 800 GiB for graphs with many edges.
2. Select GPU memory for your model and measure peak use with representative data. The source-data sizing estimates do not specify GPU memory requirements.
3. Place scratch on local SSD and retain models and accepted outputs in durable storage. Network storage and cloud block volumes are not supported as Engine working storage.
4. Size concurrent jobs for their combined resource use. Additional workers do not automatically distribute one Engine graph.

### Configure access

1. Assign the job's execution role and grant the permissions for routine scoring:

   | Resource                         | Permissions                                                               |
   | -------------------------------- | ------------------------------------------------------------------------- |
   | Source tables                    | `SELECT`                                                                  |
   | Parent databases and schemas     | `USAGE`                                                                   |
   | Query warehouse and compute pool | `USAGE`                                                                   |
   | Model                            | `READ` for bundle retrieval                                               |
   | Internal stages                  | Read and write privileges for the selected artifact and payload locations |
   | Output schema and tables         | Creation and write privileges for the selected delivery method            |

   Model `USAGE` alone does not allow downloading the bundle's files.

2. Grant provisioning roles the additional privileges to create jobs, Tasks, models and experiments. Separate these from the scoring role's routine access.

3. Configure package access through the supported repository or an external access integration.

4. Use `SnowflakeConnector.from_spcs_env()` where runtime identity is available, with the query warehouse supplied explicitly. The service execution role needs access to source data, model artifacts, output locations and tracking.

### Install dependencies

1. Follow [Snowflake setup](/sdgm/kumo-engine/get-started-with-kumo-engine/installation/snowflake-notebook) for runtime-specific installation requirements.
2. Add `kumo-engine[snowflake]` to the ML Job's Python requirements, using the approved Engine build containing the MLOps APIs. This extra supplies the Snowflake connector and SDK dependencies.
3. Add source-specific extras if the workflow accesses other services. For example, use `kumo-engine[snowflake,s3]` when it also uses the Engine S3 connector.
4. Use the same approved package version for training and scoring. Run the setup guide's import and GPU checks in the job environment and record the effective dependency versions.

## Prepare the model and prediction script

1. Select an evaluated model and retain its complete bundle, exact version and required text embedding assets. Store these outside temporary scratch. See [Publish and Retrieve Models](/sdgm/kumo-engine/user-code-guide/train-a-model/publish-and-retrieve-models).

2. Move the working prediction code into `predict_batch.py`, including imports, connector setup, graph definitions and predictive-query definitions. The [production script](/sdgm/kumo-engine/user-code-guide/deploy-a-model/overview#prepare-the-prediction-script) illustrates the structure; use its Snowflake connectors for your production schemas.

3. Supply the following inputs through your script arguments or job configuration:

   | Input               | Value                                                             |
   | ------------------- | ----------------------------------------------------------------- |
   | Model               | Exact accepted registry version or bundle path                    |
   | Data                | Source tables and recorded source version or retained snapshot    |
   | Prediction time     | Fixed UTC anchor for temporal queries; omitted for static queries |
   | Batch identifier    | One identifier for the scoring request, retained across retries   |
   | Attempt destination | A separate output table or path for each attempt                  |
   | Scratch             | Writable node-local SSD directory                                 |

   These are application inputs, not built-in Engine CLI flags. A model bundle does not contain the source data or graph snapshot.

4. Load the model and build or reload the intended snapshot. Regenerate the prediction population for the batch.

5. Call `Trainer.predict` inside the Kumo MLOps context to retain prediction records for monitoring. Configure a durable artifact root and the tracking destination for your environment.

6. Export predictions to the attempt destination. Check the saved keys, population and task-specific row counts before making the output available to the application.

## Deploy the ML Job

1. Package `predict_batch.py` and its supporting modules using the [ML Job walkthrough](/sdgm/kumo-engine/user-code-guide/deploy-a-model/overview#run-the-script-as-an-ml-job).
2. Configure the compute pool, payload stage, runtime, Python requirements and application inputs.
3. Register a reusable ML Job Definition using [Snowflake ML Jobs](https://docs.snowflake.com/en/developer-guide/snowflake-ml/ml-jobs/overview).
4. Invoke the definition once and wait for the underlying ML Job result. Review the logs and saved prediction output.
5. Create a Task or task-graph step that invokes the definition after upstream refresh. Supply the intended batch anchor and a separate attempt destination on each invocation.
6. Configure Task timeouts, failure suspension and notifications. Add bounded retries once output delivery is duplicate-safe.
7. Test that an ML Job failure makes the calling Task fail and sends the configured notification.

Snowflake registry `run_batch` provides an alternative container inference path. It writes scores without Kumo prediction-run records and requires the intended snapshot to be supplied for refreshed data. This guide uses ML Jobs calling `Trainer.predict` for Kumo monitoring.

## Run the job before scheduling

1. Run the job from a fresh runtime with the intended execution identity.
2. Review terminal job status, saved predictions and tracking records. Read the saved output through the application's access path.
3. Measure peak RAM, GPU memory, scratch use and job duration on representative data. Adjust capacity before increasing concurrency.
4. Test an interrupted run and retry before publishing its output. Keep the same model, data version, population and prediction anchor across attempts. Accept only one output for each logical batch.
5. Test rollback to the previous model version and accepted output.

## Schedule predictions and monitor results

1. Schedule the job after the upstream data refresh, or configure a dependency on the refresh task.
2. Record the model version, data or snapshot version, prediction anchor, batch identifier, accepted output location and job status together.
3. Configure operational alerts for failed jobs and missed freshness deadlines. Track job duration, resource use and requested, scored and skipped rows.
4. Compute score drift from retained Kumo prediction records. Evaluate model performance when labels become available and the prediction outcome window has completed.
5. Export monitoring results to governed tables and configure their refresh schedule, alert thresholds and responsible operator.

Job logs describe execution health; drift and labeled outcomes describe model behavior. Investigate data freshness and population changes before deciding to retrain. Registry-native scoring may use a different monitoring path from jobs calling `Trainer.predict` inside Kumo MLOps.

For output delivery, keep each attempt separate until validation succeeds. Publish the accepted location once per batch and retain the previous accepted output on failure. See [Schedule Prediction Jobs](/sdgm/kumo-engine/user-code-guide/deploy-a-model/schedule-prediction-jobs).

## Promote and refresh models

1. Train and register a candidate without changing the production alias.
2. Compare the candidate with the current model using the same evaluation population and outcome definition.
3. Run the candidate in the production job environment, then promote the accepted version. Record the exact version used for each prediction batch.
4. Retain the previous model, embedding assets and runtime configuration. To roll back, select that version for subsequent jobs and restore the previous accepted output where needed.

See [Refresh a Model](/sdgm/kumo-engine/user-code-guide/deploy-a-model/refresh-a-model) for refresh cadence and evaluation guidance.

## Assign ISV operating responsibilities

1. Assign the following responsibilities before deployment:

   | Owner                  | Responsibility                                                |
   | ---------------------- | ------------------------------------------------------------- |
   | Customer administrator | Compute, storage, networking, identities and resource grants  |
   | ISV deployer           | Compatible runtime, job code and deployment configuration     |
   | Production operator    | Schedules, alerts, promotion, rollback, retention and support |

2. For ISV-operated environments, separate customer credentials, artifact roots, outputs and execution resources. A shared Python process does not provide tenant isolation.

3. Store credentials in managed secrets or use workload identity. Define credential rotation and revocation procedures, and exclude tokens and customer rows from diagnostic logs.

4. Agree on compute ownership, artifact retention and deletion permissions.

## Troubleshoot jobs

| Symptom                              | Action                                                                                                  |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------- |
| Import or CUDA failure               | Compare the installed Engine, Python, PyTorch and CUDA versions with the selected runtime requirements. |
| Access denied                        | Review the job identity's access to the affected source, model, artifact store or output.               |
| Scratch exhaustion                   | Increase local SSD capacity or reduce overlapping jobs.                                                 |
| Job submitted but no output          | Check terminal job status, execution logs and the export destination.                                   |
| Scores exist but monitoring is empty | Check the Kumo MLOps context, artifact root and monitoring access.                                      |