> This page is for Kumo Engine.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Monitor Predictions

> Check whether a Kumo Engine prediction run's scores have shifted, and measure its accuracy once outcomes arrive

Monitor each batch of scores your model produces. When you finish this page, you can tell whether a prediction run's scores have shifted from the model's holdout scores and from the previous run, see which input columns changed since training, and measure the run's accuracy once its outcomes are known. The results appear on the prediction run in your tracker, beside its scores (on Snowflake, in a run of their own).

This page continues from [Register and Promote Models](/sdgm/kumo-engine/user-code-guide/operate-models/registry#load-a-model-by-alias), and the examples use `cfg`, `graph`, and the loaded `champion` from that page. Monitoring reads prediction runs that `trainer.predict()` stored inside a `kumo.mlops` block.

## Compare Scores With the Holdout and the Previous Run

Every prediction run that `trainer.predict()` records already compares its scores with the model's holdout scores from training, and logs the result as the `drift/psi`, `drift/ks`, and `drift/wasserstein` metrics. To also compare it with the previous run, and its input columns with the training data, call `score_drift`.

The following example scores today's snapshot and compares the run with `earlier_run`, the previous prediction run of the same model. [Find a Model's Earlier Runs](#find-a-models-earlier-runs) shows how a job loads it; on the model's first run, omit `previous=`:

```python
with kumo.mlops(cfg):
    snapshot = graph.snapshot()
    run = champion.predict(snapshot)
    drift = champion.score_drift(run, previous=earlier_run)

print("vs holdout:", drift.psi, drift.ks, drift.wasserstein)
print("vs previous run:", drift.previous_psi, drift.previous_ks, drift.previous_wasserstein)
print("row count changes:", drift.input_changes)
```

To fill `TARGET_PRED` for binary classification, which [Warehouse Monitors](/sdgm/kumo-engine/user-code-guide/operate-models/warehouse-monitors) reads, pass `binary_classification_threshold` to `predict()`, as described in [Export Evaluations](#export-evaluations).

`drift` is a `DriftReport`. The first line prints the three statistics against the holdout, such as `0.88 0.27 0.037`, and the second against the previous run. `input_changes` maps each table, and each link between tables, whose row or edge count changed since training to its count at training and now, such as `{"orders": (30000, 40000)}`. Without `previous=`, the previous run's statistics are `None`. Any statistic that cannot be computed is `None`, and `drift.reasons` says why; for example, a regression or ranking model's run cannot be compared with a run of another model version.

Inside the block, `score_drift` also writes into the prediction run: the `drift/psi_vs_prev`, `drift/ks_vs_prev`, and `drift/wasserstein_vs_prev` metrics, one table per source table under `drift/inputs/` in the run's artifacts, and a **Score drift** section in the run card. The card below belongs to a run scored on newer data than the model's training data:

![Run card of a prediction run: a headline with PSI 0.88 against the holdout, a score drift table comparing the holdout and the previous run, row and edge count changes, and notes naming the most shifted input columns](/sdgm/_fern-files/_img/20f5a3af65b3d830fb2f784a7f456fb8bc40bb2e5c88f87e934d0bde45b9267b.webp)

On Snowflake (preview), each `score_drift` and `evaluate` call inside the block records its own run in the experiment instead, named after the prediction run, such as `DRIFT_CHURN_GNN_V1_2026_11_01` or `EVAL_CHURN_GNN_V1_2026_11_01`. These runs count toward the experiment's limit of 500 runs described in [Track Runs](/sdgm/kumo-engine/user-code-guide/operate-models/track-runs#choose-a-tracker).

### Read the Drift Statistics

Each statistic compares the run's score distribution with the reference: the model's holdout scores, or the previous run's scores. The score compared is `True_PROB` for binary classification, the top class's probability for multiclass, each entity's top score for multilabel classification, ranking, and link prediction, and the predicted value for regression and forecasting.

| Statistic                   | Meaning                                                                                                                                        | How to read it                                                                                                |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| PSI (`psi`)                 | Population stability index: the scores are grouped at the reference's deciles, into up to 10 groups, and the shares in each group are compared | Rule of thumb: below 0.1 small, 0.1 to 0.25 moderate, above 0.25 large shift. The run card shows this verdict |
| KS (`ks`)                   | Kolmogorov-Smirnov statistic: the largest gap between the two cumulative distributions                                                         | From 0 (the same) to 1 (no overlap)                                                                           |
| Wasserstein (`wasserstein`) | How far the scores moved on average                                                                                                            | In score units, such as probability                                                                           |

A shift tells you the population or the data changed; it does not tell you the model got worse. Check the input tables first, then measure accuracy once outcomes arrive.

### Read the Input Column Changes

`drift.features` maps each source table and column the model uses to its statistics at training and now, with the column's PSI where one is defined, for example `drift.features["users"]["AGE"]["psi"]`.

The tables under `drift/inputs/` in the run's artifacts list the same columns, most shifted first. Each row has the column's PSI and shift verdict and the change in null rate and in distinct values; numerical columns also have the change in mean, in training standard deviations. The run card names the three most shifted columns. The `drift/inputs_max_psi` metric is the largest column PSI, so a run search such as ``metrics.`drift/inputs_max_psi` > 0.25`` finds the runs whose inputs shifted most.

They describe whole tables, so a slow change in a large event table shows up only when it is large enough to move the whole column.

## Measure Accuracy Once Outcomes Arrive

When enough time has passed for a run's outcomes to be known, call `evaluate` with a newer snapshot that contains them. For a query that predicts churn over the next 7 days, a run scored as of November 1 can be evaluated with a snapshot whose orders run past November 8.

The following example loads `run_to_check`, a prediction run scored at least one outcome window ago, and evaluates it with today's snapshot. Replace `<prediction id>` with the run's id, or find the run as in [Find a Model's Earlier Runs](#find-a-models-earlier-runs):

```python
run_to_check = kumo.PredictionResult.load_from_connector(cfg, path="predictions/<prediction id>")

with kumo.mlops(cfg):
    newer = graph.snapshot()
    evaluation = champion.evaluate(run_to_check, labels_snapshot=newer)

print(evaluation.metrics, evaluation.matured, evaluation.labelled_fraction)
```

The command prints the realized metrics, such as `{'auroc': 0.778, 'ap': 0.775}`, whether the outcome window has closed (`True`), and the share of scored rows that have an outcome, such as `1.0`. Inside the block, the metrics are logged on the evaluated prediction run as `realized/<metric>`, such as `realized/auroc`, and the run card compares each with the model's holdout value.

If the outcome window has not closed in `newer`, `evaluate` warns, returns `matured=False`, and reports metrics over the outcomes observed so far. When a metric cannot be computed, it is missing or `None` in `evaluation.metrics`, and `evaluation.reasons` says why.

`evaluate` reports the same metrics, with the same definitions, as the fit's holdout:

| Task                        | Metrics                                                                                                                |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Binary classification       | `auroc`, `ap`                                                                                                          |
| Regression and forecasting  | `mae`, `rmse`                                                                                                          |
| Multiclass classification   | `acc`                                                                                                                  |
| Multilabel classification   | `ap`, `auroc`, and `auprc`, each with `_macro` and `_micro`; `f1_micro`, `precision_micro`, `recall_micro`, `f1_macro` |
| Ranking and link prediction | `map`, `precision`, `recall`, and `f1` at each of the fit's k values                                                   |

## Find a Model's Earlier Runs

A scheduled monitoring job runs in a new process, so it first finds the runs to compare, then loads them from the artifact root as described in [Read Artifacts Back](/sdgm/kumo-engine/user-code-guide/operate-models/turn-on#read-artifacts-back).

#### MLflow, Databricks, and SageMaker

The following example finds the two most recent finished prediction runs of the `champion` model in your tracker's experiment, loads them, and compares the latest with the one before it:

```python
import mlflow

with cfg.tracking.client():
    runs = mlflow.search_runs(
        experiment_names=[cfg.tracking.experiment_name],
        filter_string=(
            "tags.`kumo.kind` = 'prediction' "
            f"and tags.`kumo.bundle_id` = '{champion.result.id}' "
            "and attributes.status = 'FINISHED'"
        ),
        order_by=["attributes.start_time DESC"],
        max_results=2,
    )

latest, *earlier = [
    kumo.PredictionResult.load_from_connector(cfg, path=f"predictions/{prediction_id}")
    for prediction_id in runs["tags.kumo.prediction_id"]
]
with kumo.mlops(cfg):
    drift = champion.score_drift(latest, previous=earlier[0] if earlier else None)
```

`latest` is the run that started last, and `earlier` holds the run before it, if any. The search needs at least one run; a newly promoted model has none until its first scoring job. Loading a run reads only its record; its scores download when a monitoring call needs them. The drift is written into the latest run, as when you call `score_drift` right after scoring.

#### Snowflake (Preview)

Snowflake runs cannot be searched by Kumo's tags, so list the prediction runs stored under the artifact root instead. With the artifact root `@ML.MODELS.KUMO/kumo`, list them in SQL:

```sql
LIST @ML.MODELS.KUMO/kumo/predictions/;
```

Each file path in the listing continues `predictions/<prediction id>/...`. Load the runs and keep those of the champion, newest first. Replace the ids with the ones the listing shows; a scheduled job runs the same `LIST` through its Snowflake session and takes the ids from the paths:

```python
prediction_ids = ["<prediction id>", "<prediction id>"]
runs = [
    kumo.PredictionResult.load_from_connector(cfg, path=f"predictions/{prediction_id}")
    for prediction_id in prediction_ids
]
latest, *earlier = sorted(
    (run for run in runs if run.bundle_id == champion.result.id),
    key=lambda run: run.manifest.created_at,
    reverse=True,
)
with kumo.mlops(cfg):
    drift = champion.score_drift(latest, previous=earlier[0] if earlier else None)
```

The unpacking needs at least one run of the champion; a newly promoted model has none until its first scoring job.

Both examples order runs by when they were scored. If a backfill scores older data after a daily run, order by prediction time instead, for queries with a time window: load more runs (with MLflow, raise `max_results`) and sort them as in the Snowflake example, but with `key=lambda run: max(run.manifest.prediction_table.anchors)`.

## Export Evaluations

To track realized accuracy over time in your warehouse, export each evaluation's rows to a table. Each row is one scored entity that has an outcome, with its scores, its prediction time, its outcome as `TARGET`, and the time it was evaluated as `EVALUATED_AT`. `dbx` is a `kumo.DatabricksConnector` for the schema that holds the table; for Snowflake, pass a `kumo.SnowflakeConnector` instead:

```python
from kumoapi.jobs import WriteMode

evaluation.export(kumo.PredictionOutputConfig(dbx, path="churn_evaluated", write_mode=WriteMode.APPEND))
```

With `WriteMode.APPEND`, each export adds its rows and the `PREDICTION_ID` and `BUNDLE_ID` columns, so one table keeps every evaluation of every model version. Exporting the same evaluation twice adds its rows twice, so a retried job should check for its `PREDICTION_ID` first. For binary classification, `TARGET_PRED` has a value only when the prediction run had a threshold: `binary_classification_threshold` passed to `trainer.predict()`, or a default threshold stored with the model. [Warehouse Monitors](/sdgm/kumo-engine/user-code-guide/operate-models/warehouse-monitors) turns this table into a monitor dashboard. To export to files instead, pass a file connector and a directory `path`; files do not support `APPEND`, so use a new directory for each evaluation.

## Next Steps

#### [Warehouse Monitors](/sdgm/kumo-engine/user-code-guide/operate-models/warehouse-monitors)

Chart exported evaluations in Databricks data quality monitoring or a Snowflake model monitor.

#### [Refresh a Model](/sdgm/kumo-engine/user-code-guide/deploy-a-model/refresh-a-model)

Decide when to retrain from drift and realized accuracy.