Hybrid Graph Neural Networks

Hybrid GNNs: Transforming Recommendation Systems with Kumo AI

Recommendation systems have been a subject of great innovation over the past few decades, starting with matrix factorization in the early 2000s and evolving into two-tower and other deep learning approaches in the 2010s. In recent years, graph neural networks emerged as the leading approach for recommender systems, enabling companies such as Pinterest, Uber, and Amazon to achieve double-digit lifts in key business metrics. Kumo offers a robust architecture known as hybrid graph neural networks (hybrid GNNs), which has been empirically shown to deliver outstanding performance on both public Kaggle data science challenges and real-world production deployments.

Understanding the complexity of recommendation systems

At their core, recommendation systems recommend content or products to users, with the goal of providing inspiration to users. However, developing these systems is inherently challenging due to fickle preferences and complex patterns of human behavior. Users vary significantly in preferences: some are explorers who constantly seek new experiences, while others are repeaters who prefer familiarity. This is made even more challenging in the face of big data, cold-start items, new users with very little interaction history, and lack of data diversity.

Because this problem is so challenging, many engineering teams have developed multi-stage recommendation pipelines involving numerous candidate generation steps followed by a complex ensemble of ranking models. These systems take tens of millions of dollars to build in human and infrastructure cost, and are challenging to maintain over time.

Introducing the hybrid GNN architecture

To address these challenges, Kumo developed a hybrid GNN approach that can create great recommendations with a single model while capturing the nuanced behaviors of different users with remarkable accuracy. The hybrid GNN models two distinct user behaviors differently within a single backbone GNN model: repeated interactions and explorative interactions, hence the name hybrid GNN. The hybrid GNN is the default model architecture for recommendation and personalization tasks at Kumo, and it can be fine-tuned to your specific dataset using the Model Planner.

Why GNNs are ideal for recommendations

GNNs are highly suitable for recommendation tasks because, unlike traditional models, they can leverage rich graph connectivity patterns to gain a deeper understanding of user preferences and surface insights that are often missed by other algorithms.

The recommendation problem forms a bipartite graph between users and items, where nodes represent the users and items, and edges represent the user-item interactions. Edges often come with timestamps. Moreover, multiple edges may exist between pairs of nodes, since a user may repeatedly interact with the same item (for example, repeat ordering of the same product in e-commerce). Given the bipartite graph of past interactions, a recommendation task can be cast as a link prediction task: one that calls for predicting future interactions between user nodes and item nodes.

Model input: Processing data with the hybrid GNN

The hybrid GNN model is designed to capture fine-grained user behaviors by leveraging graph connectivity. Similar to a standard GNN model (for example, GraphSAGE), the hybrid GNN processes input through a subgraph centered around each user node. For simplicity and efficiency, consider a 1-hop neighbor sampler:

A 1-hop neighbor subgraph contains items that a user previously interacted with, as well as features associated with the sampled users, items, and edges (for example, timestamp or price). Given the subgraph, a hybrid GNN employs a heterogeneous GNN to compute embeddings of the user and items.

Exploring the hybrid GNN model architecture

The key innovation of the hybrid GNN is its hybrid approach to computing item scores per user, which are then sorted to produce the top K item recommendations for the user. Specifically, the hybrid GNN computes item scores differently based on whether or not items are sampled within the subgraph.

These differing scoring approaches are as follows:

(1) For items sampled in the subgraph, the hybrid GNN computes item scores by applying a multi-layer perceptron (MLP) over the GNN’s item embeddings. Since the GNN’s item embedding contains information about historical interactions between the user and the item, the MLP can be applied to the item embedding to predict whether or not the user would repeat the interaction with the item.

(2) For items not sampled in the subgraph, the hybrid GNN computes item scores by taking an inner product between the GNN’s user embeddings and shallow item embeddings. As the low-dimensional item embeddings can capture similarity between items (à la matrix factorization), GNNs can use this method to recommend similar items that a user has never interacted with before.

The observation that user behaviors are diverse is the final element that makes the hybrid GNN work. Some users prefer to repeatedly interact with the same set of items (better captured by the first hybrid GNN scoring approach), while others like to constantly explore new items (better captured by the second hybrid GNN scoring approach). To accommodate such diversity across different users, the hybrid GNN learns a user-specific repetition scalar predicted from the GNN’s user embeddings with another MLP. The scalar is added to the score of the first approach to capture the repetitiveness of each user’s behavior. The more repetitive a user is, the hybrid GNN predicts a higher user-specific repetition scalar. If users are highly explorative (that is, they interact frequently with new items), the hybrid GNN predicts a lower user-specific repetition scalar.

Training and optimization: Maximizing performance with the hybrid GNN

The hybrid GNN is trained end-to-end, optimizing both types of item scores as well as the repetition scalar to maximize the predictive performance of future user-item interactions. This way, the hybrid GNN figures out the user behaviors from data on its own, producing highly accurate predictions that capture the complex nature of repetition versus exploration behaviors.

Empirical studies: Assessing hybrid GNN performance

The hybrid GNN performance was tested using a Kaggle H&M recommendation challenge. The challenge called for predicting the top 12 items each user would purchase in the next 7 days, with model performance measured by mean average precision (MAP) @ 12. The dataset contains two years of historical data consisting of 1.4M users, 106K items, and 31.7M interactions between them. The challenge attracted a total of 3,000+ teams that submitted results to the public Kaggle leaderboard over the course of the 3-month competition held in 2022.

Comparing hybrid GNN results to top Kaggle competitors

The following are the results of the hybrid GNN, evaluated on the hidden test set after the competition (Kaggle allows post-competition submissions). A comparison of the hybrid GNN results to the top Kaggle competitor submissions is provided below:

The hybrid GNN placed in the top 1% of all submissions in the Kaggle H&M recommendation challenge, which is 47% better than the median score (which is what an average data scientist can achieve with traditional techniques). Using Kumo, the entire hybrid GNN training and prediction time took approximately two hours on a single GPU, without any feature engineering required. In contrast, the leading Kaggle competition challengers utilized complex model ensembling techniques and feature engineering code that would require months to develop and maintain in production.

Ablation studies

To confirm that the hybrid GNN produces better results than traditional GNN approaches, Kumo ran an ablation study where only one ranking technique was used at prediction time. The hybrid GNN was more than 100% better than the “inner product” approach, which is the standard approach used by two-tower recommendation models.

Real-world applications: Deploying the hybrid GNN in production

Kumo has deployed the hybrid GNN model architecture to production in many enterprises, resulting in significant improvements in model performance compared to internal baselines and improvements in revenues and customer experiences. The following illustrates Kumo’s recommendation performance on a large-scale local food delivery service - with the task of recommending the restaurants that each customer will most likely order from in the next 7 days (out of 600K+ restaurants). The Kumo recommendations powered by the hybrid GNN architecture generated over $100 million in additional sales for the food delivery company.

Conclusion: The power of the hybrid GNN in recommendation systems

The hybrid GNN simplifies recommendation into a single model that adapts to diverse user behaviors, delivering accurate results without the complexity of multi-stage pipelines. By handling both repeat and exploratory interactions within one architecture, it provides a scalable, efficient solution for businesses aiming to personalize their services at enterprise scale.