Skip to navigation

Best Practices for Kumo Tabular

View as MarkdownOpen in Claude

Kumo Tabular is an in-context learning model. It does not train on your data. Instead, it reads a set of labeled context rows each time it predicts. Most of the gains come from choosing which rows it reads and from evaluating it fairly, not from tuning the model.

This guide explains how to prepare your data, evaluate predictions, and choose context and model settings for your task. If you’re new to Kumo Tabular, start with the Quickstart.

Best Practices

Keep Context and Evaluation Data Separate

Kumo Tabular does not train, so instead of a training set, split your labeled rows three ways:

  • Context pool: the rows the model draws its context from.
  • Validation set: the rows you score to compare settings.
  • Test set: the rows you set aside until you have chosen your settings. Score it once, at the end, to estimate performance on new data.

Never put validation or test rows in a context. Validation and test rows should also differ from the context pool the way your future rows will, so the scores reflect what happens in production. How should I evaluate Kumo foundation models? covers metrics and calibration.

Pick the split by what you will predict:

  • Future rows: split by time. Put the earliest period in the context pool, the next period in validation, and the latest in test. The model is then always scored on rows that come after its context.
  • New customers, products, or stores: split by group. Put all rows of each customer in one split. Otherwise the model can recognize a validation customer from that customer’s own context rows, and it scores better than it will on customers it has never seen. If you only predict for existing customers, split by time instead.
  • Rows with no time order or groups: split at random. Keep the real label mix in each split. Balancing the classes, covered later, changes only the context, not the validation or test sets.

When the label looks into the future, leave a gap between splits. For a label such as “churns in the next 30 days”, a context row from June 25 gets its label from activity up to July 25. If validation starts on July 1, that label overlaps the validation period. A gap at least as long as the label period, 30 days here, prevents this. A label about the row’s own moment, such as whether a transaction was fraud, needs no gap.

Three splits, each making a context pool, a validation set, and a test set. By time, when you predict future events from past data: along a time axis, the oldest rows form the context pool, then a gap, the validation set, another gap, and the test set with the newest rows; each gap is at least as long as the label period. By group, when you predict for customers the model has not seen: all rows of customers A to E are in the context pool, F and G in validation, and H and I in test, so no customer is in two splits. At random, when rows have no time order or groups: each split holds the same share of positive rows.

The code on this page assumes a binary classification task, with the label in a column named target that holds 0 and 1. It calls the three sets context_pool_df, validation_df, and test_df.

This helper draws a context from the pool, with the pool’s label mix by default, from a recent window, or with a chosen share of positive rows:

import warnings
import pandas as pd
def draw_context(rows, target, size, positive_share=None, seed=0, time_column=None, days=None):
"""Up to `size` rows with the natural class mix, or with the given share of positive rows."""
if time_column is not None and days is not None:
rows = rows[rows[time_column] >= rows[time_column].max() - pd.Timedelta(days=days)]
if positive_share is None:
return rows.sample(min(size, len(rows)), random_state=seed)
wanted = round(size * positive_share)
positives, negatives = rows[rows[target] == 1], rows[rows[target] == 0]
n_positive = min(wanted, len(positives))
n_negative = min(size - n_positive, len(negatives))
if n_positive < wanted or n_negative < size - wanted:
warnings.warn(f"Asked for {wanted} positive and {size - wanted} negative rows; drew {n_positive} and {n_negative}.")
context = pd.concat([positives.sample(n_positive, random_state=seed), negatives.sample(n_negative, random_state=seed)])
return context.sample(frac=1.0, random_state=seed)

On the left, the context pool along a time axis from oldest to newest rows, where only about one row in eleven is positive. A frame marks the most recent part of the pool as the recent window, set by the days argument. An arrow leads from the window to the context, a smaller set of rows in which half are positive, set by the positive_share argument. A strip beneath shows the call draw_context with size, positive_share, time_column, and days.

With time_column and days, draw_context keeps only the pool’s most recent days. With positive_share, it samples each class separately, and warns you if the pool has too few rows of a class to reach that share. The same seed draws the same rows. Improve Model Performance covers when to use each option.

Ensure No Leakage

Leakage is a feature that holds information you would not have at the moment you predict. Two common examples:

  • An outcome field. In a churn table, a cancellation date is filled in only for customers who already churned, so it gives away the label.
  • An aggregate that looks past the prediction. A customer’s total orders for the month, computed at month end, includes orders placed after a prediction made on the 15th. On the 15th, you would only know the orders up to that day.

A simple check is to see how well each feature predicts the label on its own. Compute each feature’s AUROC on the validation set. If a single feature scores above 0.95 by itself, find out why before you keep it. Often its value was recorded after the outcome. Leave ID columns out of the check, because their numbers carry no meaning.

Set the Right Column Types

Kumo Tabular processes each column according to its semantic type, using a recipe. A recipe is the set of steps that turns your columns into the inputs the model expects, and turns the model’s outputs back into predictions on your original scale. The default recipe processes numerical and categorical columns and leaves every other type out. SDM guesses each column’s type from its data type: numbers become numerical, strings become categorical, and dates become datetime. Table Semantics in the SDM guide explains the types.

TypeUse It ForWhat the Default Recipe Does
numericalOrdered quantities, such as counts, amounts, and ages, even with few distinct valuesStandardizes the column, gives each estimator one of several transforms, and softly clips extreme values, so you do not need to rescale it
categoricalValues without an order, such as plans and regions, including codes stored as numbers, and the label of a classification taskEncodes the values as numbers
datetimeDates and timestampsLeaves the column out. See Dates below.
textFree text, such as reviews and descriptionsLeaves the column out. See Text below.
idIdentifiers, such as customer and order IDsLeaves the column out, as an identifier should be

The guesses most often go wrong in four places:

  • A label stored as 0 and 1 becomes numerical, and the task turns into a regression. Mark it categorical.
  • An integer identifier, such as customer_id, becomes a number, and the model reads meaning into its size. Mark it id.
  • A code stored as a number, such as a region or product code, gets an order it does not have. Mark it categorical.
  • Free text becomes categorical, with almost every value distinct. Mark it text.

Missing values need no placeholder, because Kumo Tabular has a dedicated input for missing cells. Fill in a value only when you know what its absence means, such as a missing activity count that means zero.

import sdm
overrides = {"target": "categorical", "region_code": "categorical"}
stypes = sdm.infer_stypes(context_pool_df, text="infer", id="infer", overrides=overrides)
print(stypes)

Here, text="infer" marks string columns with many distinct, multi-word values as text, and id="infer" marks columns whose name contains the word id, such as customer_id, as id. The overrides fix types that SDM cannot guess from the values alone. In this example, they mark the 0 and 1 label as categorical, so the task is classification, and region_code as categorical, because it is a code rather than an amount. Print the result and check every column you know.

The default recipe does not use date and text columns. To use them, add them to the recipe.

Dates. AddCalendarFields turns a date into numerical fields, such as the weekday and month. Add it to the recipe, then pass recipe=recipe when you call the model:

import sdm.processing as sp
recipe = sdm.models.KumoTabular.default_recipe().prepend_features(
sp.StypeDispatch(datetime=sp.AddCalendarFields(["weekday", "month"])))

You can also build date features yourself, such as days since the last order, computed as of the moment you predict. With a split by time, use only fields that repeat, such as the weekday. A field that only marks the period, such as the month when your history covers less than a year, takes values in new rows that the context never held.

Text. Add TF-IDF features or sentence embeddings to the recipe in the same way, as Data Processing in the SDM guide describes.

Reduce Variance Between Runs

Two runs of the same setting can score slightly differently. The context is drawn at random, and each estimator reads it with its own random column order, class order, and numerical transform. An estimator is one pass of the model over the context.

This matters most when you compare settings. If the same setting moves by a point or two between runs, a one-point gain from a new setting may be chance. The steps below make your scores repeatable, so a difference you measure comes from the change you made.

From left to right: a context pool of labeled rows; context draws, one per estimator or one shared, each set by a seed that fixes which rows the context holds; estimators, each reading its context with its own column order, transform, and class order, set by the seeded generator; an average of the estimators' logits; and the prediction for each query row.

  • Fix the seeds for the context draw and the model, so the same setting gives the same score on every run.
  • Average more estimators, ideally each with its own context draw, as Choose the Number of Estimators describes.
  • Score every setting on the same validation rows, with enough positive rows for a stable metric.
  • Confirm a difference across three to five seeds. For each seed, compute a bootstrap interval for the difference from your baseline. Keep the setting only when every seed agrees on the sign and each interval excludes zero.

Improve Model Performance

The settings below can help improve your model’s performance. Their effect depends on your table, so test each one against your baseline across seeds, as Reduce Variance Between Runs describes. The first three shape the context. The context is the only information Kumo Tabular has about your table, so it usually matters most.

Balance the Classes

When the positive class is rare, as with fraud or churn, a context with the natural mix holds few positive rows. A higher share of positive rows often helps. Try a few shares on validation, such as 10%, 25%, and 50%, and choose again when you change the context size.

Favor Recent Context

If the patterns in your data change over time, for example after a pricing change or a new product launch, old rows can teach the model behavior that no longer holds. Draw the context from only the most recent rows, using the helper’s recent window, and compare a few window lengths on validation, such as the last three, six, and twelve months. A short window holds fewer rows, so check that it still has enough rows of the rarer class.

Test Larger Contexts

More context rows usually help, until the gain levels off. Test a few sizes on validation, such as 10,000, 30,000, 60,000, and 100,000 rows, and weigh any gain against GPU memory and time.

When choosing a context size, consider both rows and columns:

  • Rows: Kumo Tabular was pretrained on contexts of up to 60,000 rows. This is not a maximum context size; use validation to check whether larger contexts improve your results.
  • Columns: It was pretrained on up to 100 feature columns, and the default recipe uses at most 500 columns per estimator. Calendar fields and text features add columns, so count them too.

See What are the context-size limits for Kumo Tabular and Kumo Relational? and the GPU Sizing Guide.

Choose the Number of Estimators

Kumo Tabular averages its estimators, and the default is one. Try two. If the metric improves, try four, six, and eight. Gains beyond eight are usually small, while compute grows with every estimator.

Most of the gain comes from estimators that read different rows. By default, every estimator reads the same context. When your pool is larger than one context, give each estimator its own draw. SDM accepts one context per estimator, stacked into a single input, as its model reference describes. Convert all the draws to one table together before you split and stack them. Tables converted separately can number their categories differently and change predictions without any error.

When to Consider Fine-Tuning

Fine-tuning updates the model weights on your table. Fine-tuning gains vary by dataset. Compare the fine-tuned model with the pretrained baseline on validation data before deciding whether to use it. If you try it, do so after the steps above, and keep it only if it beats the model without fine-tuning on the same validation rows across seeds. The fine-tuning guide walks through it.

Next Steps