> This page is for Kumo Engine.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Warehouse Monitors

> Chart the realized accuracy of Kumo Engine prediction runs over time in Databricks data quality monitoring or a Snowflake model monitor

Chart how accurate your model has been over time in the monitoring your warehouse already provides. When you finish this page, the evaluations your jobs export land in a table, and a Databricks data quality monitor or a Snowflake model monitor (preview) computes accuracy metrics from it for each day.

This page continues from [Monitor Predictions](/sdgm/kumo-engine/user-code-guide/operate-models/monitor#export-evaluations): your monitoring job calls `evaluate` on a prediction run and exports the evaluation to a table with `WriteMode.APPEND`. The recipes on this page cover binary classification models.

## What the Evaluation Table Holds

Each row is one scored entity whose outcome is known. A binary classification table has these columns:

| Column             | Meaning                                                   |
| ------------------ | --------------------------------------------------------- |
| `ENTITY`           | The entity's primary key                                  |
| `ANCHOR_TIMESTAMP` | The prediction time the entity was scored as of           |
| `TARGET`           | The observed outcome                                      |
| `EVALUATED_AT`     | When `evaluate` ran                                       |
| `True_PROB`        | The predicted probability of the positive class           |
| `TARGET_PRED`      | The predicted class at the run's threshold, if it had one |
| `PREDICTION_ID`    | The prediction run's id                                   |
| `BUNDLE_ID`        | The model id, which differs for each model version        |

The Databricks monitor reads `TARGET_PRED`, so score with a threshold, as described in [Export Evaluations](/sdgm/kumo-engine/user-code-guide/operate-models/monitor#export-evaluations).

These monitors have no baseline table, so they do not compare scores with the training data. That comparison is the `drift/*` metrics on each prediction run, described in [Monitor Predictions](/sdgm/kumo-engine/user-code-guide/operate-models/monitor#compare-scores-with-the-holdout-and-the-previous-run).

## Create the Monitor

#### Databricks

The Databricks monitor is an inference log monitor from [data quality monitoring](https://docs.databricks.com/aws/en/data-quality-monitoring/). One table holds the evaluations of every model version, and the monitor computes metrics for each `BUNDLE_ID` and for all versions together.

After each evaluation, append it to a Unity Catalog table. `dbx` is a `kumo.DatabricksConnector` for the schema `main.ml`, and `evaluation` comes from `evaluate`:

```python
from kumoapi.jobs import WriteMode

evaluation.export(kumo.PredictionOutputConfig(dbx, path="churn_evaluated", write_mode=WriteMode.APPEND))
```

The export stages its rows in a volume named `_kumo_staging` in the table's schema, so the identity needs `CREATE VOLUME` on the schema, or `READ VOLUME` and `WRITE VOLUME` on that volume if it exists, plus `CREATE TABLE` on the schema.

Once the table exists, create the monitor once. Replace `<you>` with your workspace user name; the monitor's dashboard is stored under that folder:

```python
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.dataquality import (
    AggregationGranularity,
    DataProfilingConfig,
    InferenceLogConfig,
    InferenceProblemType,
    Monitor,
)

w = WorkspaceClient()
table_id = w.tables.get("main.ml.churn_evaluated").table_id
w.data_quality.create_monitor(monitor=Monitor(
    object_type="table",
    object_id=table_id,
    data_profiling_config=DataProfilingConfig(
        output_schema_id=w.schemas.get("main.ml").schema_id,
        assets_dir="/Workspace/Users/<you>/monitoring",
        inference_log=InferenceLogConfig(
            problem_type=InferenceProblemType.INFERENCE_PROBLEM_TYPE_CLASSIFICATION,
            prediction_column="TARGET_PRED",
            label_column="TARGET",
            model_id_column="BUNDLE_ID",
            timestamp_column="EVALUATED_AT",
            granularities=[AggregationGranularity.AGGREGATION_GRANULARITY_1_DAY],
        ),
    ),
))
```

`WorkspaceClient()` uses the same credentials as your notebook or job, or the `DATABRICKS_*` environment variables. For the privileges the identity needs, see the Databricks [data quality monitoring requirements](https://docs.databricks.com/aws/en/data-quality-monitoring/).

After each export, refresh the monitor:

```python
from databricks.sdk.service.dataquality import Refresh

w.data_quality.create_refresh(
    object_type="table",
    object_id=table_id,
    refresh=Refresh(object_type="table", object_id=table_id),
)
```

A refresh can take 10 to 20 minutes. When it finishes, the output schema holds `churn_evaluated_profile_metrics` and `churn_evaluated_drift_metrics`, and the dashboard under the assets folder charts them. In the profile metrics, the table-level rows (`column_name = ':table'`) hold one row per day of `EVALUATED_AT`, for each `BUNDLE_ID` and for all model versions together (`BUNDLE_ID` is `*`), with classification metrics at the run's threshold, such as `accuracy_score`, `precision`, and `recall`. The drift metrics compare each day with the day before, so they stay empty until the table holds two days of evaluations. The monitor reads only `TARGET_PRED`, so it reports no ROC AUC; use `evaluate`'s `auroc` for that.

The monitor keys each window on `EVALUATED_AT`, not on the prediction time, so a batch appears on the day you evaluated it.

#### Snowflake (Preview)

A Snowflake [model monitor](https://docs.snowflake.com/en/developer-guide/snowflake-ml/model-registry/model-observability) tracks one version of a model in the Snowflake Model Registry, such as a version that `kumo.MLOps` registered in [Register and Promote Models](/sdgm/kumo-engine/user-code-guide/operate-models/registry). It reads every row of its source table as that version's, so keep one table per version. It needs a warehouse, not a compute pool.

After each evaluation, append it to the table of the version that scored the run. `sf` is a `kumo.SnowflakeConnector` for `ML.MODELS`, and `evaluation` comes from `evaluate` on a run of version `V2`. Give the table name in upper case, so SQL can refer to it unquoted:

```python
from kumoapi.jobs import WriteMode

evaluation.export(kumo.PredictionOutputConfig(sf, path="CHURN_EVALUATED_V2", write_mode=WriteMode.APPEND))
```

Once the table exists, create the version's monitor once in the `ML.MODELS` schema. Replace `<warehouse>` with your warehouse:

```sql
CREATE MODEL MONITOR CHURN_MONITOR_V2 WITH
  MODEL=CHURN_GNN VERSION=V2 FUNCTION=predict
  SOURCE=CHURN_EVALUATED_V2 WAREHOUSE=<warehouse>
  REFRESH_INTERVAL='1 day' AGGREGATION_WINDOW='1 day'
  TIMESTAMP_COLUMN=ANCHOR_TIMESTAMP ID_COLUMNS=('ENTITY')
  PREDICTION_SCORE_COLUMNS=('True_PROB') ACTUAL_CLASS_COLUMNS=('TARGET');
```

The role needs `CREATE MODEL MONITOR` and `CREATE TABLE` on the schema. Column names are string literals in their stored case, such as `('True_PROB')`. The statement returns `Model monitor service CHURN_MONITOR_V2 successfully created.` When you promote a new version, create its table and monitor the same way.

The monitor refreshes on its interval and keys each window on `ANCHOR_TIMESTAMP`, the prediction time. Without a baseline, the monitor reports realized quality only. Read its metrics in SQL. Quote the score column inside the column argument:

```sql
SELECT * FROM TABLE(MODEL_MONITOR_PERFORMANCE_METRIC(
  'CHURN_MONITOR_V2', 'ROC_AUC', '1 DAY', TO_TIMESTAMP_TZ('2026-10-01'), TO_TIMESTAMP_TZ('2026-11-01')));
SELECT * FROM TABLE(MODEL_MONITOR_STAT_METRIC(
  'CHURN_MONITOR_V2', 'COUNT', '"True_PROB"', '1 DAY', TO_TIMESTAMP_TZ('2026-10-01'), TO_TIMESTAMP_TZ('2026-11-01')));
```

The first query returns one row per day of prediction time with the ROC AUC of that day's rows, about the same value as `evaluate`'s `auroc` for the run, and the second the number of rows. Drift functions such as `MODEL_MONITOR_DRIFT_METRIC` need a baseline and fail without one.

## Next Steps

#### [Monitor Predictions](/sdgm/kumo-engine/user-code-guide/operate-models/monitor)

Check score drift and input changes on each prediction run.

#### [Refresh a Model](/sdgm/kumo-engine/user-code-guide/deploy-a-model/refresh-a-model)

Decide when to retrain from drift and realized accuracy.