How should I evaluate Kumo foundation models?

Predict on held-out examples whose labels are excluded from the context. For temporal tasks, use only records and labels available at each prediction time.

Compare with your existing model or another baseline on the same examples. Choose metrics that reflect the decisions the predictions support. Check probability calibration if those decisions depend on probability values, and measure latency and memory use at the expected context size and scoring volume.

Public Benchmark Datasets

Choose tasks supported by your model and runtime, and use the benchmark’s prescribed splits and evaluator.

For Kumo Relational, RelBench v1 and RelBench v2 additional datasets provide multi-table databases and prediction tasks. Follow the RelBench evaluation protocol for splits and scoring.

For Kumo Tabular, use:

• TabArena: benchmark tasks, results, and links to supporting resources.
• TALENT datasets: classification and regression datasets, with setup instructions in the repository.
• BeyondArena: datasets with random, temporal, and grouped splits.