Skip to navigation

In-Context Learning

View as Markdown

Kumo Tabular and Kumo Relational are part of a special class of foundation models called “in-context learning” models. Unlike traditional models where a training dataset is used to fit a tree or MLP, in-context learning uses alternating layers of attention to directly make a prediction from a set of context examples. The model itself is only trained once, and can be reused for many datasets.

In-context learning models are typically trained on millions of simulated datasets, which makes them highly robust to messy or missing data, and also very good at making an accurate prediction from a small set of context examples.

Kumo Tabular makes predictions over a single table, and Kumo Relational uses a set of connected tables.

Single-Table In-Context Learning

Kumo Tabular is an all-purpose model that can do binary and multiclass classification and regression over a single table of data.

More specifically, it can impute missing values of prediction rows by using a set of several thousand context rows as an input to the model.

Unlike traditional ML where a model would be fitted over a training dataset, and then used for inference, in-context learning does all computation at inference time.

The context and the rows to predict are fed to the model at the same time, and the prediction is made via many layers of alternating row and column attention that identify latent patterns in the data, similar to how an LLM operates over a stream of text. Kumo Tabular uses the column, row and in-context attention introduced by TabPFN (Hollmann et al., 2023), TabICL (Qu et al., 2025) and TabICLv2 (Qu et al., 2026), and the Kumo Tabular launch post describes its architecture in full.

Three panels. Context input: a Task column of two known labels and one unknown is attached by dotted lines to the matching rows of a table of features, with the labelled rows marked as context rows and the last as the prediction row. Reason within the table: the labels are added to the first column of the table, drawn as a block of rows by columns by embedding depth, with column attention connecting the rows and row attention connecting the columns. In-context learning: each row is compressed to one embedding, cross-sample attention lets the prediction row read the context rows, and the result is a predicted label.

As a result of being pre-trained on such a wide variety of simulated input datasets, the model is highly effective at identifying natural patterns in the data (e.g., cause and effect) and as a result can get significantly better accuracy than traditional ML models that are only trained for a single task.

Scatter plot of TabArena Elo against median inference time per thousand rows on a log scale. The Kumo Tabular frontier, from Kumo Tabular-S with one estimator to Kumo Tabular-L, runs along the top of the chart, above other tabular foundation models and well above LightGBM in its default, tuned, and tuned and ensembled settings.

Accuracy against prediction time on TabArena, run on a single GPU. Every Kumo Tabular size is more accurate than LightGBM at any setting, and Kumo Tabular-S is also faster. From the Kumo Tabular launch post.

Relational Foundation Models

Kumo Relational is an all-purpose model that can do classification, regression, forecasting and recommendations over a set of connected tables, such as users, orders and products.

More specifically, you describe what to predict in a predictive query, such as which users will churn next month, and the model gathers its own context examples from the history of the database, much like a retrieval-augmented LLM gathers documents before it answers a question.

Unlike traditional ML, where a data scientist first flattens the tables into one row per entity with hand-built aggregates such as “average order value in the last 30 days”, Kumo Relational works directly on the tables, their foreign keys and their timestamps.

Each context example is a small subgraph around an entity at a point in the past, such as a user with their orders, paired with what actually happened next. Every example only sees data from before its own point in time, which prevents data leakage. Attention then runs within each table, as in Kumo Tabular, and across tables along their foreign keys. This builds on relational deep learning (Fey et al., 2024) and KumoRFM-2, and the Kumo Relational launch post describes the architecture in full.

Three panels. Context input: a Task column of two known targets and one unknown attaches to rows of a Users table, each user owns rows in an Orders table, and each order points to a row in an Items table. Reason within tables: the targets are added to the first column of each table, drawn as blocks with column attention connecting rows and row attention connecting columns. Reason across tables: each table becomes a slab of row embeddings, graph attention carries information along foreign keys from orders to users and items, and cross-sample attention connects the Task and Users rows through a readout.

After KumoRFM-2, Figure 3.

As a result of being pre-trained on a wide variety of simulated databases, the model learns to filter, correlate and aggregate information across tables on its own, and beats both supervised relational deep learning models and gradient-boosted trees on expert-engineered features, on databases it has never seen.

Scatter plot of RelArena-α Elo across tasks against total runtime on a log scale. Kumo Relational sits on the Pareto front, close in Elo to the most accurate system while running about 430 times faster, and well above LightGBM and supervised relational models such as GraphSAGE and RelGNN.

Accuracy against total runtime on RelArena-α. Kumo Relational is about 430 times faster than the most accurate system at similar accuracy, and both faster and far more accurate than LightGBM. From the Kumo Relational launch post.

Pretraining on Simulated Data

Both models are pre-trained entirely on simulated data, much like self-driving cars. Real driving data rarely includes the edge cases a car has to handle, so carmakers use simulators to generate them. Real tables have the same gap, and unlike the text that LLMs learn from, there is no web-scale supply of them.

More specifically, each simulated table comes from a structural causal model, a random graph of cause and effect between hidden variables. Some variables become columns and one becomes the target, while the rest stay hidden, like the unmeasured causes behind real data.

Each variable is computed from its causes by a randomly chosen function, such as a small neural network or a tree, and missing values and noisy rows are added on purpose because real tables are messy. Because the simulator is a procedure rather than a fixed dataset, it can produce an endless supply of tables, each with new causes and effects.

On every table, the model sees most of the rows with their labels as context and learns to predict the rest, which is exactly the task it later performs on your data. The Kumo Tabular launch post describes the simulator in full, and the Kumo Relational launch post shows how it extends to whole databases.

Three panels. Causal graph: a random graph of hidden variables in which each node applies a randomly chosen function such as a small neural network, a Gaussian process, a tree or a linear map. Some nodes become feature columns x1 to x4, one becomes the target y, and the rest stay hidden or unused. Simulated table: the resulting table with numerical and categorical columns, a target column and missing values. Training task: the first rows keep their labels as context rows and the rest become prediction rows, repeated for 35 million to 137 million tables.

After 35 million to 137 million such tables, depending on model size, the model carries a strong prior about how cause and effect show up in rows and columns. That prior is why it can beat models that only learn from the data you give them, and why it handles missing or corrupt values without any cleanup.

Kumo Tabular

Predict on a single table from labelled example rows.

Kumo Relational

Predict across related tables using a predictive query.

Temporal Hetero-GNN

The supervised architecture Kumo Engine fits per task.

KumoRFM Research

Research background on relational foundation models.