When developing a new predictive query for decision-making in your organization, you may need to answer the question “how much benefit does this predictive query bring, compared to the alternatives?” for your stakeholders. The alternative could be a simple business rule or heuristic, or an existing machine learning model.

Kumo provides tools and best practices to make this comparison straightforward.

Comparing Against Historical Baselines

Often, the most impact comes from going from 0 to 1: deploying machine learning to a business process where none existed before. For example, you might be building a new email notification that recommends products for a customer to buy.

In this situation, compare your machine learning model against a simple historical baseline. For product recommendation, a historical baseline could be the “top 10 most popular products on your platform” or “the last 5 items that the user viewed”.

Kumo can automatically generate many simple historical baselines for your predictive query, and compare its performance against them using metrics such as Accuracy and Recall. For more complex historical baselines, download the holdout dataset from the Kumo UI, or generate predictions on your own holdout dataset and load it into a notebook or spreadsheet for custom analysis.

Comparing Against Existing Models

If you are replacing an existing production model with a predictive query (whether to reduce maintenance cost or improve performance with deep learning), you need to compare both models before proceeding.

To do this, hold out a portion of your labeled data, generate predictions using both models, and compare standard evaluation metrics such as AUROC. The following guidelines help you avoid common evaluation setup mistakes that can produce incorrect or confusing results.

• Use Kumo’s model planner to define the holdout time split: The simplest way to define a common holdout dataset is by specifying a time range. Using a single line of code in the Kumo model planner, you can train on data up to a specific date and hold out data after that date. After the model is trained, Kumo automatically generates evaluation metrics on the held-out dataset. Assuming you have correctly specified the time_col for all tables, Kumo’s trainer ignores all held-out data during training, so you do not have to worry about data leakage from the future. Remember to also check your comparison model for data leakage.

• Bring your own holdout dataset: If you prefer not to use Kumo’s built-in time split feature, physically separate your training data from your holdout data before Kumo ingestion. Train your model using Kumo on your training dataset, then generate batch predictions on a graph containing the training and holdout dataset. You can then calculate evaluation metrics manually by comparing your batch predictions against your ground-truth labels. This approach gives you more confidence that the Kumo model is not leaking information from the future. However, it is more prone to human error: you must hold out data correctly across all tables in the graph. If you are doing this for the first time, share the full holdout dataset with your Kumo contact so they can verify the setup.

• Perform sanity checks on the holdout data: Skipping this step can completely invalidate your results. Look for the following issues:

  • Duplicate rows in the holdout dataset
  • Major distribution shifts from train to holdout
  • Outliers that cause significant variance in evaluation metrics

• Remember that Kumo needs raw data: GNNs need access to raw data to perform well. If you use Kumo on pre-engineered features, you will match the performance of classical models such as GBDT at best, because feature engineering drops information that Kumo can no longer learn from.

Learn More

• Data Preparation Best Practices
• Model Planner and AutoML
• Evaluation
• Training