RelBench

Why benchmarks matter

Every breakthrough in machine learning has been preceded by a benchmark. ImageNet transformed computer vision, GLUE and SuperGLUE accelerated NLP, and OGB catalyzed graph-learning research. These benchmarks define a common evaluation setting, make results reproducible, and give researchers a shared language for comparing methods.

Relational databases are the backbone of enterprise data, with customers linked to transactions, transactions linked to products, and products linked to categories. Before RelBench, there was no standard benchmark for evaluating ML methods on this multi-table data. Researchers used different datasets, preprocessing, train-test splits, and metrics, making results across papers difficult to compare.

The problem with earlier evaluation

Earlier work on ML for relational data often used proprietary data or ad-hoc extracts that discarded the rest of the database. Even when studies started from the same source data, different joins, feature engineering, and temporal cutoffs meant that their results were not directly comparable.

Random train-test splits also permit training on future data when predicting past events. That data leakage does not reflect production conditions, where only prior information is available at prediction time.

What is RelBench?

RelBench is a NeurIPS 2024 benchmark for deep learning on relational databases. It provides seven databases across diverse domains, 30 realistic prediction tasks, standardized temporal train-validation-test splits, and baseline implementations for reproducible comparison.

Across the benchmark, there are 51 tables, more than 103 million rows, and 489 columns. The datasets come from real-world sources including Amazon product reviews, H&M retail transactions, Stack Overflow, Formula 1 racing, clinical trials, online classified ads, and an event platform.

The design preserves full relational structure, enforces temporal integrity, spans varied domains and scales, and makes data loading, preprocessing, training, and evaluation code available for replication.

Relational deep learning

RelBench evaluates relational deep learning (RDL), which converts a relational database into a heterogeneous graph. Each table row becomes a node, and each primary-foreign key relationship becomes an edge. A GNN learns over that graph together with a deep tabular model for initial node features.

This differs from a traditional workflow where a data scientist manually joins tables, engineers features, and trains a flat-table model such as XGBoost or LightGBM.

The seven databases

• rel-amazon includes products, reviews, and ratings, with tasks such as negative-review prediction, rating forecasting, and recommendations.
• rel-avito models ads, users, search queries, and contextual features, with tasks such as ad-click and top-seller prediction.
• rel-event models users, events, RSVPs, and metadata to test attendance and engagement prediction at scale.
• rel-f1 links drivers, constructors, races, qualifying sessions, pit stops, lap times, and results. Its dense structure makes it challenging despite its smaller row count.
• rel-hm contains customers, articles, and transactions, with churn and recommendation tasks.
• rel-stack includes users, posts, comments, votes, tags, badges, and post history, testing multi-hop reasoning.
• rel-trial contains 15 clinical-trial tables, including studies, conditions, interventions, outcomes, sponsors, facilities, and eligibility criteria.

Task types

RelBench defines 30 tasks that are tied to a database, an entity table, and a target column or relation.

For classification and regression, labels must come from before the validation cutoff. For recommendation, test-window interactions are the ground truth and models can only use prior interactions.

Baselines and results

RelBench compares three approaches to prediction over relational databases:

1. LightGBM with automated features uses an automated featurization pipeline with little manual effort.
2. An expert data scientist with LightGBM manually creates joins, aggregations, and transformations for each task.
3. GraphSAGE operates on the database represented as a heterogeneous graph, learning features end to end without manual engineering.

The paper’s user study found that the expert workflow took an average of 12.3 hours and 878 lines of code per task. The GraphSAGE pipeline required about 30 minutes of setup and 56 lines of code, a reported 24-fold reduction in time and 15-fold reduction in code.

The GNN baseline consistently matched or outperformed the expert workflow on most tasks, while the automated LightGBM baseline struggled more with multi-hop relationships.

How relational deep learning works

The RDL baseline transforms database structure into a learned prediction pipeline.

1. Database to graph: Every row becomes a node and every primary-foreign key link becomes an edge.
2. Initial node features: A deep tabular model encodes numerical, categorical, text, and timestamp values into node representations.
3. Message passing: GraphSAGE aggregates information from neighboring rows across multiple hops.
4. Prediction: A task-specific head supports classification, regression, or ranking.

Each message-passing round is analogous to one SQL join in the source article’s explanation. The difference is that the GNN learns useful aggregations from the relational neighborhood instead of requiring a person to specify every join and aggregate.

When training graphs are constructed, RelBench only includes timestamped rows before the relevant cutoff and removes edges to future rows. This maintains temporal integrity at the graph level.

Why RelBench matters

RelBench lets researchers evaluate methods on the same databases, tasks, temporal splits, and metrics. It also quantifies the human effort of traditional relational ML, creating a basis for judging both model quality and engineering cost.

The benchmark creates a foundation for work on relational foundation models, including methods that may generalize across databases and prediction tasks rather than training from scratch for every task.

Open challenges include scaling beyond the largest benchmark database, few-shot and zero-shot prediction, reasoning over much more complex schemas, and handling continuously changing production graphs.

Further reading

• RelBench paper
• RelBench project
• RelBench GitHub repository
• KumoRFM paper