> This page is for Kumo Engine. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server. # MLOps Reference > Artifact root layout, manifest and lineage fields, drift statistics, evaluation columns, custom trackers, progress events, and limits for Kumo Engine MLOps Look up the exact layout, fields, and definitions behind Kumo Engine MLOps. The task pages, starting with [Turn On MLOps](/sdgm/kumo-engine/user-code-guide/operate-models/turn-on), show how to use these features; this page describes what they write and how their numbers are computed. ## Artifact Root Layout Each stored artifact has its own directory under the artifact root, with a `_provenance.json` manifest at the place the tree shows. [Manifests and Lineage](#manifests-and-lineage) describes the manifests. ```text / ├── snapshots/// │ ├── kumo_snapshot.json │ └── / │ ├── index/ │ └── _provenance.json ├── training_tables/tt-/ │ ├── part-*.parquet │ ├── _provenance.json │ └── _SUCCESS ├── models// │ ├── model/ │ ├── holdout.parquet │ ├── metrics.json │ ├── training-spec.json │ └── _provenance.json ├── predictions// │ ├── predictions/ │ │ ├── part-*.parquet │ │ ├── _provenance.json │ │ └── _SUCCESS │ └── skipped/ └── assets/// └── _provenance.json ``` | Directory | Contents | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `snapshots/` | One directory per graph and data version. `kumo_snapshot.json` names the current generation, which holds the graph index | | `training_tables/` | The training table's rows as Parquet parts | | `models/` | The model bundle: weights and encoders in `model/`, the holdout predictions in `holdout.parquet`, the holdout metrics, and the training specification. The bundle holds only the model's own state; its manifest records what else it needs to predict | | `predictions/` | One directory per prediction run, never overwritten. `predictions/` holds the scores. `skipped/` exists when the run skipped rows, and lists each skipped row with the reason it was skipped. A run that also wrote embeddings has an `embeddings/` directory | | `assets/` | Files that models need besides their bundle, such as word vectors for text columns. Each is stored once, by its SHA-256, and shared by every model that uses it | ## Identifiers | Identifier | Names | Where to find it | | -------------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `snapshot.id`, `fit_result.snapshot_id` | A graph snapshot, as `@` | `snapshots///` | | `training_table.id` | A training table, starting with `tt-` | `training_tables//` | | `fit_result.id` | A model, by its bundle id: 64 hexadecimal characters | `models//`; the `kumo.bundle_id` run tag; `kumo_bundle_id` on the registered version; the `BUNDLE_ID` column of an appended table | | `prediction.id` | A prediction run | `predictions//`; the `kumo.prediction_id` run tag; the `PREDICTION_ID` column of an appended table | | `fit_result.registered_model.version` | A registered version | The registry | | `fit_result.tracking`, `prediction.tracking` | The tracker's run, with `platform`, `experiment`, `run_id`, `run_name`, and `url` | The tracker | A training table's id is the same each time the same snapshot, query, generation plan, split seed, and engine build produce it. A model's bundle id is a hash of the bundle's contents. ## Manifests and Lineage Each `_provenance.json` is a JSON manifest that records what the artifact is and where it came from. It contains metadata only. The same manifest is available in Python as `snapshot.manifest`, `training_table.manifest`, `fit_result.manifest`, and `prediction.manifest`. Every manifest has these fields: | Field | Meaning | | ---------------- | ----------------------------------------------------------------------------------------------------------- | | `kind` | `snapshot`, `training_table`, `model`, `prediction`, or `asset` | | `schema_version` | The manifest format version | | `id` | The artifact's identifier | | `created_at` | When the artifact was created, in UTC | | `environment` | The Kumo Engine, DiskGraph, PyTorch, and Python versions and the engine build that created it | | `location` | Where the artifact is stored | | `parents` | The manifests of the artifacts it was made from, embedded in full, so one manifest traces the whole lineage | The `parents` and other fields of each kind: | Kind | `parents` | Other fields | | -------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Snapshot | `source_tables`: each table's connector, location, and version | `build`: when the snapshot was built, how long it took, and its size on disk | | Training table | `snapshot`, and for a table read from a source, `source_table` | `query`, `generation_plan`, `task_type`, `split_seed`, `build` | | Model | `training_table`, and `warm_start` for a model trained from another model | `fit` (start, end, and seconds), `fit_mode`, `tracking` (the fit run), `dependencies`, `holdout` | | Prediction run | `model` and the scoring `snapshot` | `prediction_table`, `scoring`, `outputs`, `status`, `timings`, `stats` | The model and prediction run fields in more detail: | Field | Meaning | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `dependencies` | What `predict` needs besides the bundle and a snapshot: the predictor entry point, the `kumo-engine` package with any extras such as `text`, and each asset with its kind, SHA-256, and location | | `holdout` | The holdout scores: `rows`, whether they were `truncated` to fit a size limit, their exact `counts` in fixed bins, and the label `window`. Score drift compares against these counts | | `prediction_table` | The query's SHA-256, the `anchors` (prediction times), `requested_rows`, `scored_rows`, `skipped_rows`, a sample of up to 20 skipped ids, and `skipped_location` | | `scoring` | The binary classification threshold and the number of classes returned, each with where it came from, such as `bundle` | | `outputs` | Each destination the run wrote: the `output_config` destination, and with MLOps on, the copy under the artifact root | | `stats` | Per output, the row count, and with MLOps on, the scores' exact counts in fixed bins | An embedded parent leaves out its mutable fields and statistics: a model's `tracking` and `holdout`, and a prediction run's `outputs`, `stats`, `status`, and `timings`. A parent that has no manifest, such as a bundle from a release before MLOps, appears as a stub with its id only. `graph.snapshot(record_sources=False)` records each source table without its location: only the connector, the table name, and a SHA-256 of the location. ## Drift Statistics [Monitor Predictions](/sdgm/kumo-engine/user-code-guide/operate-models/monitor) describes how to run `score_drift`. These are the definitions behind its numbers. ### Score Drift Score drift compares the distribution of a prediction run's scores with a reference distribution. It is computed twice: * **Against the holdout**: the model's own holdout scores at training. Logged on the prediction run as `drift/psi`, `drift/ks`, and `drift/wasserstein`. * **Against a previous run**: with `score_drift(run, previous=earlier)`. Logged as `drift/psi_vs_prev`, `drift/ks_vs_prev`, and `drift/wasserstein_vs_prev`. | Statistic | Definition | Range | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `psi` | Population stability index: the sum of (p − q) × ln(p / q) over up to 10 groups of bins, where p and q are the run's and the reference's shares. The groups are cut where the reference's cumulative share reaches each tenth. Shares are floored at 0.000001 | 0 for identical distributions. Rule of thumb: below 0.1 is a small shift, 0.1 to 0.25 moderate, above 0.25 large | | `ks` | Kolmogorov Smirnov (KS) statistic: the largest gap between the two cumulative distributions, at the bin edges | 0 for the same distribution to 1 for disjoint ones | | `wasserstein` | Wasserstein-1 distance: how far the scores moved on average, computed over the bin midpoints, with the end bins closed at each side's recorded minimum and maximum | In score units | The score each statistic uses depends on the task: | Task | Score | Bins | | ------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | Binary classification | `True_PROB` | 100 bins of width 0.01, plus one for a score of exactly 1 | | Multiclass classification | The top class's probability | The same probability bins | | Multilabel classification | Each entity's top score | The same probability bins | | Ranking, link prediction | Each entity's top score | The holdout's top score percentiles, from the 0th to the 100th | | Regression, forecasting | The prediction | The training target's histogram bins, or 50 equal bins from its minimum to its maximum, with its quartiles as extra edges | Each statistic is computed exactly from the counts that both sides stored on the same bins. A statistic is absent, with a note, when the two sides were counted on different bins. Probability bins are the same for every model, but the bins of a ranking, link prediction, regression, or forecasting model come from that model, so a previous run of another version of such a model cannot be compared. Link prediction holdout scores are partial, so its numbers against the holdout carry no shift verdict. ### Input Columns `score_drift` also compares each column the model encodes with the training data, using the column statistics stored with the training and scoring snapshots. It reads no data again. A tracked `score_drift` writes one table per source table, `drift/inputs/.json`, with one row per column, the most shifted first: | Column | Column types | Meaning | | --------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `psi` | Numerical, categorical | PSI against the training snapshot. Numerical: over up to 10 groups of fixed value bins, exact. Categorical: over the categories, approximate when the category counts were capped or sampled | | `shift` | Numerical, categorical | The PSI rule of thumb: small, moderate, or large | | `mean Δ sd` | Numerical | The change in the mean, in training standard deviations | | `null Δ pp` | All | The change in the share of nulls, in percentage points | | `distinct Δ≈ %` | Numerical, categorical, text | The approximate change in distinct values, accurate to within a few percent | | `newest Δ days` | Timestamp | How much later the newest value is now | | `tokens Δ≈ %` | Text | The change in the average number of tokens per value, in percent, from a sample of about 10,000 rows | A table leaves out measures that none of its columns has, and a column whose type changed shows only the old and new type. `drift/inputs_max_psi` is the largest column PSI, which you can use in alerts, for example ``metrics.`drift/inputs_max_psi` > 0.25``. These statistics describe whole snapshots, so a slow change in a large event table is diluted. ### Input Sizes `drift/table_rows` lists each source table whose row count changed since the training snapshot, with its rows at training, rows now, and the change in percent. `drift/link_edges` lists each edge, such as `orders.USER_ID → users`, whose edge count changed. `DriftReport.input_changes` holds the same counts in Python. ## Evaluation Columns `evaluation.export()` writes one row per scored entity whose outcome is known. [Warehouse Monitors](/sdgm/kumo-engine/user-code-guide/operate-models/warehouse-monitors) uses these columns. For binary classification: | Column | Meaning | | ---------------------------- | ---------------------------------------------------------------------------------------------------------- | | `ENTITY` | The entity's primary key | | `ANCHOR_TIMESTAMP` | The prediction time | | `TARGET` | The realized outcome | | `EVALUATED_AT` | When the evaluation ran, in UTC | | `True_PROB` | The predicted probability | | `TARGET_PRED` | The predicted label, which needs a threshold | | `PREDICTION_ID`, `BUNDLE_ID` | The prediction run and the model, added only when you export to a table with `write_mode=WriteMode.APPEND` | ## Custom Trackers To record runs in another system, pass any object with three methods as `tracking=` to `kumo.MLOps`. Kumo builds a `RunRecord` for each run, with the same run name, parameters, metrics, tags, and summaries every tracker receives: ```python import kumo.engine as kumo from kumo.engine.tracking import Event, RunRecord class PrintTracker: def start(self, record: RunRecord) -> None: print("start", record.name) def on_trial(self, record: RunRecord, event: Event) -> None: print("trial", event.trial, event.name) def finish(self, record: RunRecord | None, error: BaseException | None) -> None: if record is not None: print("finish", record.name, record.metrics) cfg = kumo.MLOps("s3:///kumo", tracking=PrintTracker()) ``` A fit inside `with kumo.mlops(cfg):` prints `start fit-users-binary_classification-