Skip to navigation
User Code GuideOperate Models

Warehouse Monitors

View as Markdown

Chart how accurate your model has been over time in the monitoring your warehouse already provides. When you finish this page, the evaluations your jobs export land in a table, and a Databricks data quality monitor or a Snowflake model monitor (preview) computes accuracy metrics from it for each day.

This page continues from Monitor Predictions: your monitoring job calls evaluate on a prediction run and exports the evaluation to a table with WriteMode.APPEND. The recipes on this page cover binary classification models.

What the Evaluation Table Holds

Each row is one scored entity whose outcome is known. A binary classification table has these columns:

ColumnMeaning
ENTITYThe entity’s primary key
ANCHOR_TIMESTAMPThe prediction time the entity was scored as of
TARGETThe observed outcome
EVALUATED_ATWhen evaluate ran
True_PROBThe predicted probability of the positive class
TARGET_PREDThe predicted class at the run’s threshold, if it had one
PREDICTION_IDThe prediction run’s id
BUNDLE_IDThe model id, which differs for each model version

The Databricks monitor reads TARGET_PRED, so score with a threshold, as described in Export Evaluations.

These monitors have no baseline table, so they do not compare scores with the training data. That comparison is the drift/* metrics on each prediction run, described in Monitor Predictions.

Create the Monitor

The Databricks monitor is an inference log monitor from data quality monitoring. One table holds the evaluations of every model version, and the monitor computes metrics for each BUNDLE_ID and for all versions together.

After each evaluation, append it to a Unity Catalog table. dbx is a kumo.DatabricksConnector for the schema main.ml, and evaluation comes from evaluate:

from kumoapi.jobs import WriteMode
evaluation.export(kumo.PredictionOutputConfig(dbx, path="churn_evaluated", write_mode=WriteMode.APPEND))

The export stages its rows in a volume named _kumo_staging in the table’s schema, so the identity needs CREATE VOLUME on the schema, or READ VOLUME and WRITE VOLUME on that volume if it exists, plus CREATE TABLE on the schema.

Once the table exists, create the monitor once. Replace <you> with your workspace user name; the monitor’s dashboard is stored under that folder:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.dataquality import (
    AggregationGranularity,
    DataProfilingConfig,
    InferenceLogConfig,
    InferenceProblemType,
    Monitor,
)
w = WorkspaceClient()
table_id = w.tables.get("main.ml.churn_evaluated").table_id
w.data_quality.create_monitor(monitor=Monitor(
    object_type="table",
    object_id=table_id,
    data_profiling_config=DataProfilingConfig(
        output_schema_id=w.schemas.get("main.ml").schema_id,
        assets_dir="/Workspace/Users/<you>/monitoring",
        inference_log=InferenceLogConfig(
            problem_type=InferenceProblemType.INFERENCE_PROBLEM_TYPE_CLASSIFICATION,
            prediction_column="TARGET_PRED",
            label_column="TARGET",
            model_id_column="BUNDLE_ID",
            timestamp_column="EVALUATED_AT",
            granularities=[AggregationGranularity.AGGREGATION_GRANULARITY_1_DAY],
        ),
    ),
))

WorkspaceClient() uses the same credentials as your notebook or job, or the DATABRICKS_* environment variables. For the privileges the identity needs, see the Databricks data quality monitoring requirements.

After each export, refresh the monitor:

from databricks.sdk.service.dataquality import Refresh
w.data_quality.create_refresh(
    object_type="table",
    object_id=table_id,
    refresh=Refresh(object_type="table", object_id=table_id),
)

A refresh can take 10 to 20 minutes. When it finishes, the output schema holds churn_evaluated_profile_metrics and churn_evaluated_drift_metrics, and the dashboard under the assets folder charts them. In the profile metrics, the table-level rows (column_name = ':table') hold one row per day of EVALUATED_AT, for each BUNDLE_ID and for all model versions together (BUNDLE_ID is *), with classification metrics at the run’s threshold, such as accuracy_score, precision, and recall. The drift metrics compare each day with the day before, so they stay empty until the table holds two days of evaluations. The monitor reads only TARGET_PRED, so it reports no ROC AUC; use evaluate’s auroc for that.

The monitor keys each window on EVALUATED_AT, not on the prediction time, so a batch appears on the day you evaluated it.

Next Steps

Monitor Predictions

Check score drift and input changes on each prediction run.

Refresh a Model

Decide when to retrain from drift and realized accuracy.