Recommender Systems with LLMs

This guide examines how encoder-based LLM embeddings and graph transformers work together for personalized recommendation systems. On the H&M fashion recommendation benchmark, the source reports that graph-based models substantially outperform an LLM-only baseline, while LLM embeddings provide an additional uplift when used as graph features.

The recommendation problem

Recommendation systems drive revenue across e-commerce, streaming, advertising, and social media. The core task is to predict which items a user will interact with next. In practice, it requires two kinds of signal: what items are - their text descriptions, categories, and attributes - and how users behave through purchase sequences, browsing patterns, and co-purchase relationships.

Large language models produce semantic embeddings that capture meaning, context, and nuance in product text. Graph neural networks capture behavior through direct interactions, indirect connections between similar users, and temporal changes in preferences. This research asks whether combining both produces better recommendations. It uses H&M’s public Kaggle fashion-recommendation dataset, which includes customer profiles, transaction histories, and detailed article descriptions across three connected tables.

Why LLMs alone fall short

An LLM can encode a product’s name, description, color, and material as a dense vector that captures semantic similarity. Cotton T-shirts will be near other cotton T-shirts, and evening dresses near formal wear. But personalization requires behavioral patterns: which customers buy which products, in what order, how often, and how those patterns relate to other customers. LLM embeddings encode what products are, not what customers do.

Encoder-based LLMs for embeddings

Encoder-based LLMs convert text into continuous vector representations called embeddings. Unlike generative models that produce text token by token, encoder models process the full input and output a fixed-size vector with context-sensitive semantic information.

The source evaluates two encoders:

• OpenAI text-embedding-3-large - a commercial embedding model for semantic similarity.
• intfloat/e5-base-v2 - an open-source sentence transformer intended to offer competitive embeddings at lower cost.

For recommendations, the product name, detailed description, color, and material are concatenated into one string and passed through an encoder. The resulting vector positions semantically similar products near each other in embedding space.

The LLM-only baseline

In the LLM-only baseline, product embeddings come directly from item text. Each customer is represented by the average embedding of their previously purchased products, and recommendations are the products closest to that customer vector.

This reduces each customer to a centroid of their history. A customer who bought running shoes and a formal suit receives a representation between athletic wear and formalwear that may describe neither interest well. Averaging also loses ordering, frequency, and distinct purchase modes.

The graph transformer approach

Kumo’s graph transformer models the problem as a heterogeneous temporal graph. Customers, transactions, and products are nodes, while purchases are timestamped edges. The graph records not only what was bought but also when, by whom, and in what context.

Why graphs capture what LLMs miss

Graph structure supports multi-hop reasoning. If Customer A bought products that Customer B also bought, and B recently bought a new item, message passing can propagate that signal as a recommendation for A. The temporal graph can also distinguish seasonal preferences and purchase frequencies. These signals are encoded in the graph rather than approximated with manual aggregations.

Kumo’s baseline text handling

Without LLM integration, Kumo uses GloVe embeddings for text columns. GloVe averages word-level representations, which captures basic semantic content but not the contextual meaning of a full phrase. For example, “lightweight waterproof running jacket” becomes an average of word vectors rather than a context-aware representation.

Combining LLMs with graph transformers

The integration pipeline uses LLM-generated embeddings as input features to the graph transformer. Rather than treating text understanding and behavioral modeling as separate systems, the combined approach feeds semantic item representations into the graph network.

How the integration works

Kumo’s Sequence data type can use encoder-produced text vectors as features.

1. Text and categorical columns in the articles table - product name, description, color, material, shape, and type - are concatenated for each product.
2. An OpenAI or Hugging Face encoder pre-encodes the string as a dense vector.
3. Kumo ingests the vector as a Sequence-type feature on the article node.
4. The graph transformer uses that feature during message passing with relational structure.

The predictive query

The task is to predict the 12 distinct articles each customer is most likely to purchase in the next seven days. This ranking objective drives graph construction, training, and evaluation, while the LLM vectors remain enriched node features alongside other signals.

H&M dataset and experiment design

The H&M Personalized Fashion Recommendations dataset is a public Kaggle competition dataset with three linked tables:

• Customers - profile data including demographic and membership attributes.
• Transactions - timestamped purchase history linking customers and articles.
• Articles - product name, detailed description, color, material, shape, type, and other categorical or text attributes.

Four experimental configurations

The LLM-only baseline computes OpenAI embeddings for products and averages each customer’s purchased-product vectors. The Kumo-only baseline uses the full heterogeneous temporal graph with GloVe text encoding. The combined configurations replace GloVe with LLM encoders and keep the other graph architecture components identical.

Evaluation metrics

All models recommend the top 12 articles most likely to be purchased in the next seven days.

• MAP@12 measures ranking quality by rewarding correct items placed higher in the list.
• Precision@12 is the share of recommended items that were purchased.
• Recall@12 is the share of actual purchases that appeared in the top 12.
• F1@12 is the harmonic mean of precision and recall.

Results: 15x graph advantage, 11% LLM uplift

The benchmark shows two findings: graph-based approaches greatly outperform LLM-only recommendations, and LLM embeddings improve the graph baseline across every reported metric.

Finding 1: graphs outperform LLMs by 15x

The LLM-only approach reaches MAP@12 of 0.00190, while the Kumo graph-only approach reaches 0.02856 - a 15x improvement in ranking quality. The source reports that the LLM-only baseline is 2x to 40x worse across the other metrics. It cannot capture collaborative-filtering signals, temporal dynamics, or multi-hop transaction-graph relationships.

Finding 2: LLM embeddings improve graph models by 4-11%

Replacing GloVe with OpenAI embeddings improves the graph model from 0.02856 to 0.02976 MAP@12, a 4.2% improvement. Precision@12 rises 11.3%, recall@12 rises 8.3%, and F1@12 rises 10.6%. The Hugging Face e5-base-v2 model also improves the graph baseline by roughly 4% to 7% across metrics, according to the source.

The improvement is not a replacement for behavioral modeling. LLM encoders provide richer contextual item representations, which the graph transformer then propagates across customer-product interactions.

Practical implications

The source’s conclusion is a hierarchy: LLMs alone are insufficient for personalized recommendations, graphs alone are strong, and graphs enhanced with LLM features are strongest.

Do not use LLMs as standalone recommendation engines

The 15x gap reflects an architectural mismatch rather than marginal tuning. Semantic item similarity does not represent collaborative or temporal behavior.

Invest in graph infrastructure first

The source’s largest gain is from moving from LLM-only recommendations to a heterogeneous temporal graph. Teams should prioritize relational structure before refining text encoders.

Layer LLM embeddings on top for incremental gains

Once the graph is in place, modern LLM embeddings can replace simpler text encoders with no changes to graph construction, training objectives, or serving architecture.

Stronger encoders produce better results

OpenAI text-embedding-3-large outperforms e5-base-v2 in the reported combined configurations. The text encoder can be upgraded independently while graph architecture remains the behavioral foundation.