> This page is for Kumo Engine.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Track Runs

> Record Kumo Engine fits and prediction runs in MLflow or Snowflake, and read what each run shows

Record each fit and prediction run in the experiment tracker your team already uses. When you finish this page, every fit appears as a run with its model plan as parameters, per-epoch training curves, holdout metrics, and a run card, and you can find a model's runs with a filter.

This page continues from [Turn On MLOps](/sdgm/kumo-engine/user-code-guide/operate-models/turn-on), which covers the install extras for each tracker. You choose the tracker with the `tracking` argument of `kumo.MLOps`; the rest of your code stays the same.

## Choose a Tracker

Each tab shows the `tracking` value for one platform and what that platform needs before the first run. Pass it as `kumo.MLOps(<artifact root>, tracking=tracking)`. Kumo creates the experiment if it does not exist.

#### MLflow Server

Record runs on any MLflow tracking server. Replace `<mlflow-host>` with your server's address; `"churn"` is the experiment name:

```python
from kumo.engine.integrations.mlflow import MlflowTracking

tracking = MlflowTracking("churn", target="http://<mlflow-host>:5000")
```

The process that runs your code needs network access to the server and, unless the server proxies artifact uploads, write access to the server's artifact store. Without `target`, `MlflowTracking` records wherever MLflow itself is configured to record, for example through the `MLFLOW_TRACKING_URI` environment variable or `mlflow.set_tracking_uri()`. With `target`, the experiment name is required.

#### Databricks

Record runs in a Databricks workspace's MLflow by passing a `DatabricksConnector` as the target. The connector supplies the workspace and its credentials; use the same connector settings as for your Databricks data. Replace `<warehouse-id>` with a SQL warehouse ID and `<you>` with your workspace user name, usually your email address:

```python
import kumo.engine as kumo
from kumo.engine.integrations.mlflow import MlflowTracking

dbx = kumo.DatabricksConnector(catalog="main", schema_name="ml", warehouse_id="<warehouse-id>")
tracking = MlflowTracking("/Users/<you>/churn", target=dbx)
```

The experiment name is a workspace path, its parent folder must already exist, and the identity needs permission to create and edit experiments there. Without an identity, the connector uses the notebook or job credentials, a Databricks configuration profile, or the `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID`, and `DATABRICKS_CLIENT_SECRET` environment variables. To use a service principal explicitly, pass `identity=kumo.DatabricksIdentity(host="https://<workspace-host>", client_id="<client-id>", client_secret="<client-secret>")` to the connector. See [Connect Your Data: Databricks](/sdgm/kumo-engine/user-code-guide/connect-your-data-databricks) for connector settings.

#### SageMaker

Record runs on a SageMaker managed MLflow tracking server or MLflow App by passing its ARN, which the server's or app's details page in the SageMaker console shows:

```python
from kumo.engine.integrations.mlflow import MlflowTracking, SageMakerMLflow

tracking = MlflowTracking("churn", target=SageMakerMLflow("<tracking-server-or-app-arn>"))
```

A stopped tracking server must be running before you create `kumo.MLOps`; start it with `aws sagemaker start-mlflow-tracking-server --tracking-server-name <name>` and wait until it reports `Started`. `SageMakerMLflow` uses your default AWS credentials, such as a notebook or instance role. To assume another role, pass `SageMakerMLflow("<arn>", identity=kumo.S3Identity(role_arn="<role-arn>"))`.

The identity needs `sagemaker-mlflow:*` on the tracking server ARN, or `sagemaker:CallMlflowAppApi` on the MLflow App ARN, plus read and write access to the server's S3 artifact store and to your artifact root.

MLflow's printed run links do not open on SageMaker, so `fit_result.tracking.url` is `None`; use `fit_result.tracking.run_name` to find the run. Open the MLflow UI from SageMaker Studio (Applications, then MLflow), or create a temporary sign-in URL with `aws sagemaker create-presigned-mlflow-tracking-server-url --tracking-server-name <name>`. For an MLflow App, use `aws sagemaker create-presigned-mlflow-app-url --arn <app-arn>`.

#### Snowflake (Preview)

Record runs in Snowflake Experiment Tracking with a `SnowflakeConnector`. Replace `<warehouse>` with your warehouse; runs are stored in the `ML.MODELS` schema:

```python
import kumo.engine as kumo
from kumo.engine.integrations.snowflake import SnowflakeTracking

sf = kumo.SnowflakeConnector(warehouse="<warehouse>", database="ML", schema_name="MODELS")
tracking = SnowflakeTracking(sf, experiment_name="CHURN")
```

The connector's role needs `CREATE EXPERIMENT` on the schema, or `MODIFY` on an existing experiment. Pass `database=` and `schema_name=` to `SnowflakeTracking` to keep runs in another schema. Runs appear in Snowsight under **AI & ML**, then **Experiments**.

Snowflake records the same metrics, run card, and lineage, with these differences:

* Kumo tags are stored as run parameters.
* Hyperparameter search trials are recorded inside the fit run, not as child runs.
* Runs list no dataset inputs and no system metrics.
* `cfg.tracking.client()` is not available.
* A run keeps at most 200 metric names, and an experiment at most 500 runs. Kumo warns at 450 runs and stops tracking at 500, so use one experiment per model.

## Read a Fit Run

Run a fit inside the block, then open the run from `fit_result.tracking.url` or from the experiment's run list. Recent MLflow versions open the UI on GenAI pages; select **Model training** in the sidebar, or add `?workflowType=machine_learning` to the URL.

The run list below shows two fits and a prediction run of the `users-churn` model, with the datasets each run used and the registered version each fit produced. Kumo describes a new experiment with the filters for its fits and prediction runs:

![MLflow run list for the churn experiment, showing fit-users-churn-v1, fit-users-churn-v2 and a prediction run, with dataset and registered model columns](/sdgm/_fern-files/_img/e9f1979369fe797e29c8136738d2db228874d92817bfec2703113db24d3987da.webp)

### Run Names

| Run                                     | Name                                                                                        |
| --------------------------------------- | ------------------------------------------------------------------------------------------- |
| Fit that registered a version           | `fit-<model>-v<version>`                                                                    |
| Fit without a registry                  | `fit-<entity table>-<task type>-<YYYYMMDD-HHMMSS>`, using the fit's start time in UTC       |
| Prediction run of a registered version  | `predict-<model>-v<version>-<prediction date>`                                              |
| Prediction run of an unregistered model | `predict-<entity table>-<task type>-<first 8 characters of the model id>-<prediction date>` |

`<model>` is the registered model name; for Unity Catalog, it is the last part of the name. A prediction run of a static query, which has no prediction time, adds the run's start time, as `-<HHMMSS>`, after the date. Snowflake uses the same names in upper case with underscores, such as `PREDICT_CHURN_GNN_V1_2026_11_02`. A Snowflake fit always keeps its start time, as in `FIT_CHURN_GNN_20261105_142037`, because Snowflake runs cannot be renamed; its registered version is in the `kumo.model_version` parameter.

To name runs yourself, pass `run_name=` to the tracker, for example `MlflowTracking("churn", target=..., run_name="weekly")`. Fits are then named `fit-weekly` and prediction runs `predict-weekly-<prediction date>`.

### Run Card

The run's description, shown on the MLflow **Overview** tab, starts with a run card. It leads with the tuned holdout metric and says which trial and epoch the reported metrics come from. It then lists the holdout metrics, the classification matrix at the threshold when a binary model has one, the training data per split, data preparation time and size, lineage from source tables to model, and the location of every artifact.

![Run card of fit-users-churn-v1 with the holdout AUROC, the predictive query, a holdout metrics table, rows per split, and data preparation times](/sdgm/_fern-files/_img/d17382556284ee55ebd3db17d97c29e23806938c8fb546bfc02dd787298af1e3.webp)

Kumo writes and rewrites the card. In MLflow, the card ends with the line "kumo rewrites everything above this line; editing it stops kumo's updates.", a horizontal line, and a hidden marker that the description editor shows as `<!-- /kumo:card sha256=... -->`. Write your own notes after that marker and keep the marker. If you edit anything above it, or remove it, Kumo stops updating the card. A Snowflake (preview) run card is written once when the run ends and has no notes line.

### Metrics and Parameters

| Name                             | What it records                                                                                    |
| -------------------------------- | -------------------------------------------------------------------------------------------------- |
| `train/<metric>`, `val/<metric>` | Per-epoch training and validation metrics, with the epoch as the step                              |
| `test/<metric>`                  | Holdout metrics of the model the bundle ships, such as `test/auroc`, `test/ap`, and `test/logloss` |
| `system/<metric>`                | CPU, memory, disk, network, and, with a GPU, GPU usage during the run (MLflow only)                |
| Parameters                       | The model plan's options, under their model plan names, such as `training_job.num_experiments`     |

In Python, the same holdout metrics are in `fit_result.metrics` with an underscore, such as `fit_result.metrics["test_auroc"]`. When the model plan refits the best model on training and validation data after tuning, the refit model is the one that ships: `test/<metric>` describes it, `fit_result.metrics` reports it as `test_refit_<metric>`, and the tracker logs the model before the refit as `before_refit/<metric>`.

The **Model metrics** tab charts each group.

![Model metrics tab of a fit run with test metrics as bars and train metric curves over two epochs](/sdgm/_fern-files/_img/b18e52e64a24ce54cad297fb8ffac3f891d7bc84c7821b6cc012c01af5f5c63b.webp)

`loss` is the training objective of the model plan, so compare it only between fits with the same objective. To compare different models, use the tuned metric, or `test/logloss` for classification. A model loaded from the registry has no `test/logloss`, as described in [Promote a Version](/sdgm/kumo-engine/user-code-guide/operate-models/registry#promote-a-version).

### Tags and Inputs

Every run carries `kumo.kind` (`fit`, `trial`, or `prediction`) and `kumo.engine_version`. A fit also carries `kumo.bundle_id`, `kumo.task_type`, and `kumo.snapshot_id`, and, when it registered a version, `kumo.model_name` and `kumo.model_version`. A prediction run carries `kumo.prediction_id`, `kumo.bundle_id` of the model it scored with, and, when that model was loaded from the registry, `kumo.model_name` and `kumo.model_version`; a model loaded by path leaves them out. `kumo.model_name` holds the full registered name, such as `main.ml.users_churn` on Unity Catalog. In MLflow, a fit lists the training table, the graph snapshot, and each source table as dataset inputs, and a hyperparameter search records each trial as a child run.

## Find Runs

Use these filters in the MLflow run search box, or as `filter_string` in `mlflow.search_runs()`. Tag keys that contain a dot need backticks. Filters by model name need a registry, as described in [Register and Promote Models](/sdgm/kumo-engine/user-code-guide/operate-models/registry):

| To find                                   | Filter                                                                         |
| ----------------------------------------- | ------------------------------------------------------------------------------ |
| All fits                                  | ``tags.`kumo.kind` = 'fit'``                                                   |
| All fits of a registered model            | ``tags.`kumo.kind` = 'fit' and tags.`kumo.model_name` = 'users-churn'``        |
| All prediction runs of a registered model | ``tags.`kumo.kind` = 'prediction' and tags.`kumo.model_name` = 'users-churn'`` |
| All prediction runs of one model id       | ``tags.`kumo.kind` = 'prediction' and tags.`kumo.bundle_id` = '<model id>'``   |

To call MLflow yourself with the tracker's server and credentials, use the tracker's client. This example lists the fits in the tracker's experiment:

```python
with cfg.tracking.client() as client:
    experiment = client.get_experiment_by_name(cfg.tracking.experiment_name)
    fits = client.search_runs(
        [experiment.experiment_id],
        filter_string="tags.`kumo.kind` = 'fit'",
    )
print(len(fits), "fits")
```

Inside the `with` block, `client` is an `MlflowClient` for the tracker's server, and module-level calls such as `mlflow.search_runs()` reach the same server.

## Next Steps

#### [Register and Promote Models](/sdgm/kumo-engine/user-code-guide/operate-models/registry)

Register each fit in Unity Catalog, SageMaker MLflow, or the Snowflake Model Registry, and promote a version.

#### [Training Diagnostics](/sdgm/kumo-engine/user-code-guide/train-a-model/training-diagnostics)

Interpret training metrics and holdout predictions in more depth.