MLOps Reference

Look up the exact layout, fields, and definitions behind Kumo Engine MLOps. The task pages, starting with Turn On MLOps, show how to use these features; this page describes what they write and how their numbers are computed.

Artifact Root Layout

Each stored artifact has its own directory under the artifact root, with a _provenance.json manifest at the place the tree shows. Manifests and Lineage describes the manifests.

<artifact root>/
├── snapshots/<graph id>/<data version>/
│   ├── kumo_snapshot.json
│   └── <generation>/
│       ├── index/
│       └── _provenance.json
├── training_tables/tt-<id>/
│   ├── part-*.parquet
│   ├── _provenance.json
│   └── _SUCCESS
├── models/<model id>/
│   ├── model/
│   ├── holdout.parquet
│   ├── metrics.json
│   ├── training-spec.json
│   └── _provenance.json
├── predictions/<prediction id>/
│   ├── predictions/
│   │   ├── part-*.parquet
│   │   ├── _provenance.json
│   │   └── _SUCCESS
│   └── skipped/
└── assets/<asset kind>/<sha256>/
    └── _provenance.json

Identifiers

A training table’s id is the same each time the same snapshot, query, generation plan, split seed, and engine build produce it. A model’s bundle id is a hash of the bundle’s contents.

Manifests and Lineage

Each _provenance.json is a JSON manifest that records what the artifact is and where it came from. It contains metadata only. The same manifest is available in Python as snapshot.manifest, training_table.manifest, fit_result.manifest, and prediction.manifest.

Every manifest has these fields:

The parents and other fields of each kind:

The model and prediction run fields in more detail:

An embedded parent leaves out its mutable fields and statistics: a model’s tracking and holdout, and a prediction run’s outputs, stats, status, and timings. A parent that has no manifest, such as a bundle from a release before MLOps, appears as a stub with its id only.

graph.snapshot(record_sources=False) records each source table without its location: only the connector, the table name, and a SHA-256 of the location.

Drift Statistics

Monitor Predictions describes how to run score_drift. These are the definitions behind its numbers.

Score Drift

Score drift compares the distribution of a prediction run’s scores with a reference distribution. It is computed twice:

• Against the holdout: the model’s own holdout scores at training. Logged on the prediction run as drift/psi, drift/ks, and drift/wasserstein.
• Against a previous run: with score_drift(run, previous=earlier). Logged as drift/psi_vs_prev, drift/ks_vs_prev, and drift/wasserstein_vs_prev.

The score each statistic uses depends on the task:

Each statistic is computed exactly from the counts that both sides stored on the same bins. A statistic is absent, with a note, when the two sides were counted on different bins. Probability bins are the same for every model, but the bins of a ranking, link prediction, regression, or forecasting model come from that model, so a previous run of another version of such a model cannot be compared. Link prediction holdout scores are partial, so its numbers against the holdout carry no shift verdict.

Input Columns

score_drift also compares each column the model encodes with the training data, using the column statistics stored with the training and scoring snapshots. It reads no data again. A tracked score_drift writes one table per source table, drift/inputs/<table>.json, with one row per column, the most shifted first:

A table leaves out measures that none of its columns has, and a column whose type changed shows only the old and new type. drift/inputs_max_psi is the largest column PSI, which you can use in alerts, for example metrics.`drift/inputs_max_psi` > 0.25. These statistics describe whole snapshots, so a slow change in a large event table is diluted.

Input Sizes

drift/table_rows lists each source table whose row count changed since the training snapshot, with its rows at training, rows now, and the change in percent. drift/link_edges lists each edge, such as orders.USER_ID → users, whose edge count changed. DriftReport.input_changes holds the same counts in Python.

Evaluation Columns

evaluation.export() writes one row per scored entity whose outcome is known. Warehouse Monitors uses these columns. For binary classification:

Custom Trackers

To record runs in another system, pass any object with three methods as tracking= to kumo.MLOps. Kumo builds a RunRecord for each run, with the same run name, parameters, metrics, tags, and summaries every tracker receives:

import kumo.engine as kumo
from kumo.engine.tracking import Event, RunRecord
class PrintTracker:
    def start(self, record: RunRecord) -> None:
        print("start", record.name)
    def on_trial(self, record: RunRecord, event: Event) -> None:
        print("trial", event.trial, event.name)
    def finish(self, record: RunRecord | None, error: BaseException | None) -> None:
        if record is not None:
            print("finish", record.name, record.metrics)
cfg = kumo.MLOps("s3://<bucket>/kumo", tracking=PrintTracker())

A fit inside with kumo.mlops(cfg): prints start fit-users-binary_classification-<time>, then finish with the fit’s holdout metrics, such as test/auroc.

record.summaries holds the fit’s tables, notes, and datasets. Each dataset has a name, a source (connector, location, version, and version_kind), a schema, a profile, and a context, its role. Skip summary types your tracker does not handle; new types may be added.

A tracker can define these optional members:

If a tracker method raises, Kumo warns and stops calling the tracker for that fit, except for one final finish(None, None).

Progress Events

trainer.fit(callbacks=[...]) calls each callback with every kumo.engine.tracking.Event. Each event has a name, a time, and, where they apply, a trial and epoch counted from 1, metrics, and params:

trial_ended and fit_ended carry params["status"] and, on failure, params["error"]. A callback that raises is logged and not called again. To write each event as a JSON line, pass callbacks=[lambda event: print(json.dumps(event.to_dict()))].

graph.snapshot(callbacks=[...]) reports snapshot_started, then table_staged for each table, then snapshot_ended with snapshot_id and reused. Each carries params["seconds"], the time since the snapshot started. With callbacks=[print], each event prints as its name and fields:

snapshot_started graph=<graph id> tables=['users', 'items', 'orders'] seconds=0.0
table_staged table=users seconds=0.0
table_staged table=items seconds=0.0
table_staged table=orders seconds=0.0
snapshot_ended snapshot_id=<graph id>@<data version> reused=False seconds=1.3

Limits

• Tracking time. At the end of a fit, Kumo waits at most 30 seconds for tracking to finish, and up to one hour for model logging and registration.
• MLflow HTTP calls. During a fit, MLflow calls use 2 retries and a 30 second timeout, and model uploads a 600 second timeout. Set MLFLOW_HTTP_REQUEST_MAX_RETRIES and MLFLOW_HTTP_REQUEST_TIMEOUT to change them.
• Metric names. Names with @ are logged with _at_, so map@10 becomes test/map_at_10, and acc is logged as accuracy. Parameters that MLflow rejects are counted in the kumo.dropped_params tag.
• Snowflake (preview). A run keeps at most 1,000 parameters, and the metric name limit listed in Track Runs, where a per-epoch series counts as one name. Parameters and metrics over the limits are counted in kumo.dropped_params and kumo.dropped_metrics.

Next Steps