> This page is for Kumo Engine.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# MLOps Overview

> Store, track, and register every Kumo Engine model and prediction run in the MLOps tools you already use

Turn on MLOps to keep a record of every model your Kumo Engine code trains and every batch of predictions it makes. Kumo stores each artifact under one location you choose, records each training and prediction run in your experiment tracker, and, when you name a registry, registers each trained model there. With these records you can answer which model and which data produced a table of scores, compare versions of a model, and promote a new version to production.

MLOps requires Kumo Engine 1.1.0 or newer.

You keep your existing code. You create one `kumo.MLOps` configuration and run your workflow inside `with kumo.mlops(cfg):`. This section continues from [Training](/sdgm/kumo-engine/user-code-guide/train-a-model/training): you need a graph, a predictive query, and a model plan that train successfully.

## What Kumo Records

With MLOps turned on, the calls you already make store their results under an **artifact root**, such as `s3://<bucket>/kumo`:

| Call                              | What is stored                                         |
| --------------------------------- | ------------------------------------------------------ |
| `graph.snapshot()`                | The graph snapshot                                     |
| `query.generate_training_table()` | The training table                                     |
| `trainer.fit()`                   | The model bundle                                       |
| `trainer.predict()`               | The prediction run: its scores and the rows it skipped |

Each stored artifact also records where it came from, so a model leads back to its training table, graph snapshot, and source table versions.

The **experiment tracker** records each fit and each prediction run as a run, with parameters, metrics, and a run card that summarizes the result. When you name a **model registry**, it receives each trained model as a new version, and Kumo moves an alias, such as `candidate`, to it.

## How the Pieces Fit Together

A model moves through the same loop on every platform:

```mermaid
flowchart LR
    F["<b>Fit</b><br />Store the model<br />Record a fit run"] --> R["<b>Register</b><br />New version<br />alias candidate"]
    R --> P["<b>Promote</b><br />Move alias champion<br />to a better version"]
    P --> S["<b>Score</b><br />Load champion<br />Record a prediction run"]
    S --> M["<b>Monitor</b><br />Compare scores<br />Realized accuracy"]
    M --> F
```

1. **Fit.** `trainer.fit()` inside the block stores the model and records the fit, with its per-epoch curves and holdout metrics.
2. **Register.** With a registry configured, the fit registers the model as a new version and moves your registration alias to it.
3. **Promote.** You compare the new version with the one in production and move the serving alias, such as `champion`, when it is better.
4. **Score.** A scheduled job loads the model by its serving alias and calls `trainer.predict()`, which stores and records the prediction run, as described in [Score in Batch](/sdgm/kumo-engine/user-code-guide/operate-models/score).
5. **Monitor.** You compare a prediction run's scores with the model's holdout scores and with earlier runs, and measure realized accuracy once outcomes are known, as described in [Monitor Predictions](/sdgm/kumo-engine/user-code-guide/operate-models/monitor).

## Supported Platforms

| Platform            | Experiment tracking                                          | Model registry                      |
| ------------------- | ------------------------------------------------------------ | ----------------------------------- |
| Self-managed MLflow | Any MLflow tracking server                                   | The server's MLflow Model Registry  |
| Databricks          | MLflow in the Databricks workspace                           | Unity Catalog                       |
| Amazon SageMaker    | SageMaker managed MLflow: a tracking server or an MLflow App | The same server's or app's registry |
| Snowflake (preview) | Snowflake Experiment Tracking                                | Snowflake Model Registry            |

Snowflake support is in preview: it uses Snowflake ML APIs that are themselves in preview, and its behavior can change.

The artifact root can be on local or mounted disk, Amazon S3, Google Cloud Storage, a Unity Catalog volume, or a Snowflake internal stage. The artifact root and the tracker do not need to be on the same platform; for example, a fit on your own GPU machine can store its model in S3 and record its run in Databricks.

## Next Steps

To run fits and scoring as scheduled jobs, start from a recipe: [Databricks Jobs](/sdgm/kumo-engine/user-code-guide/operate-models/databricks), [SageMaker Pipelines](/sdgm/kumo-engine/user-code-guide/operate-models/sagemaker), or [Snowflake Tasks (preview)](/sdgm/kumo-engine/user-code-guide/operate-models/snowflake). [MLOps Reference](/sdgm/kumo-engine/user-code-guide/operate-models/reference) lists the stored layout, manifest fields, and drift definitions.

#### [Turn On MLOps](/sdgm/kumo-engine/user-code-guide/operate-models/turn-on)

Choose an artifact root, wrap your workflow in a block, and read stored artifacts back.

#### [Track Runs](/sdgm/kumo-engine/user-code-guide/operate-models/track-runs)

Record fits and prediction runs in MLflow or Snowflake and read what each run shows.

#### [Register and Promote Models](/sdgm/kumo-engine/user-code-guide/operate-models/registry)

Register each fit, promote a version to production, and load a model by its alias.

#### [Score in Batch](/sdgm/kumo-engine/user-code-guide/operate-models/score)

Score a registered model with `trainer.predict()` so every batch is recorded.

#### [Monitor Predictions](/sdgm/kumo-engine/user-code-guide/operate-models/monitor)

Compare scores with the holdout and earlier runs, and measure realized accuracy.

#### [Warehouse Monitors](/sdgm/kumo-engine/user-code-guide/operate-models/warehouse-monitors)

Chart exported evaluations in Databricks or Snowflake monitoring.