> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server. # Relational Graph Transformers > How graph-aware attention overcomes message-passing limits for relational databases and supports relational foundation models **Source title:** Relational Graph Transformers: A New Frontier in AI for Relational Data\ **Published:** April 28, 2025\ **Authors:** Federico Lopez, Matthias Fey, and Jure Leskovec ## From GNNs to graph transformers Graph Neural Networks (GNNs) learn from relational data by treating rows as nodes, foreign keys as edges, and propagating messages along those connections. An enterprise database with `customers`, `transactions`, and `products` becomes a heterogeneous graph where every row is a node and every foreign-key relationship is an edge. This captures relational structure without manual feature engineering, but deeper multi-hop dependencies expose structural constraints in message passing. Message-passing GNNs aggregate information from immediate neighbors one hop at a time. Relational Graph Transformers are designed to overcome the resulting limits. The progression resembles the shift from recurrent neural networks to transformers in natural-language processing. Where recurrent models process text sequentially, transformers let every token attend directly to every other token. Relational Graph Transformers apply that principle to relational data: nodes can interact directly regardless of graph distance. **Analogy:** A message-passing GNN is a game of telephone, where information travels through intermediaries and can become distorted. A graph transformer is a conference call, where participants can speak directly to one another. ## Why message-passing GNNs hit a wall ![Comparison of message-passing GNNs, standard transformers, and relational graph transformers, including their strengths and limitations.](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nvidia-sdgm.docs.buildwithfern.com/a7dac1f844c494dcc9edc33834f9553ee58679db2aeb4aaa4ee27da81c3daf61/img/research/relational-graph-transformers-approach-comparison.svg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260920T164627Z&X-Amz-Expires=604800&X-Amz-Signature=c30820573f61e4117bdec9495b64bbcb84eb1203f8261698797b1f4ed26c56dc&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) Three related issues constrain message-passing GNNs on relational data: the multi-hop bottleneck, over-squashing, and limited expressiveness. ### The multi-hop bottleneck Related entities in a database can be several hops apart. Two transactions from the same customer communicate through the customer node: transaction A -> customer -> transaction B. Each GNN layer extends the receptive field by one hop, so practical 2-to-4-layer GNNs have a limited effective range. Adding layers also increases computation and parameters and can make training less stable. ### Over-squashing Even when a GNN reaches distant nodes, it must compress their information through intermediate nodes into fixed-size vectors. A customer connected to transactions, products, categories, and reviews can become a bottleneck where subtle predictive combinations are lost. **Analogy:** Asking librarians to summarize every section in one sentence and then asking a head librarian to summarize those sentences loses important detail. This is over-squashing. ### Limited expressiveness Standard message-passing GNNs are bounded by the 1-Weisfeiler-Leman graph-isomorphism test. Some non-isomorphic graph structures therefore receive the same representation, making particular relational patterns invisible to the model. The multi-hop bottleneck, over-squashing, and the 1-WL bound stem from the same choice: aggregating strictly along edges one hop at a time. ## From database to graph ![Database-to-graph conversion pipeline: represent each row as an entity, derive a schema graph from tables and foreign keys, then form a relational entity graph.](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nvidia-sdgm.docs.buildwithfern.com/5ccfac0d10bbe8708712a10e2c6f7ed834f0e3d42d72f22c66c76a2f3d6984db/img/research/relational-graph-transformers-database-pipeline.svg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260920T164627Z&X-Amz-Expires=604800&X-Amz-Signature=8c3b57e22e6b29cb46ee6302f699d995a3046d8a2658f20e13ee16d46bab890e&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) Before a Relational Graph Transformer operates on enterprise data, the database is converted into a schema-preserving graph representation. ### 1. Entity representation Each row becomes an entity with a primary key, foreign keys, descriptive attributes, and optional timestamps. A transaction row, for example, has its own ID, a foreign key to its customer, attributes such as amount and payment method, and an event timestamp. ### 2. Schema graph Each table becomes a node type, while each foreign-key relationship becomes an edge type. The schema graph is the blueprint for the full entity graph. ### 3. Relational entity graph Every row becomes a concrete node of its table type, and every foreign-key value creates an edge to the referenced row. The result is a heterogeneous graph that preserves the distinction between node types and relationship types. The source describes this conversion as lossless: rows, column values, foreign-key relationships, and timestamps remain available in the graph representation. ### Handling multi-modal attributes The architecture uses specialized encoders before fusing modalities into unified node features. * **Numerical features** pass through normalized MLPs. * **Categorical variables** use learnable embedding tables. * **Text fields** use pre-trained language models such as Sentence-BERT. * **Images** use pre-trained CNNs or vision transformers. * **Timestamps** can be categorical, continuous, or cyclic features. PyTorch Frame fuses these modality embeddings into row vectors that serve as initial transformer node features. ## The Relational Graph Transformer architecture ![Relational Graph Transformer pipeline: encode multi-modal features, sample a two-hop subgraph, apply relation-aware attention, and produce a task-specific prediction.](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nvidia-sdgm.docs.buildwithfern.com/9202e87d1e98a75d1081986ef631cc6392a38ac6e1a71532e4760a5d19b4c4ed/img/research/relational-graph-transformers-architecture-pipeline.svg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260920T164627Z&X-Amz-Expires=604800&X-Amz-Signature=761f70c116190c6147f7e9d739a3b396d6900897000b6b248787a02b35dc3099&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) A naive transformer over every node in a database graph ignores topology and requires O(n²) attention over millions of nodes. Relational Graph Transformers instead combine graph-aware attention, relation-aware edge encoding, and scalable subgraph sampling. ### Graph-aware attention Attention incorporates relational topology directly. Nodes prioritize local neighbors while retaining the ability to capture long-range dependencies. Graph structure biases attention toward nearby nodes, but the model can learn to prioritize distant signals when useful. ### Relation-aware edge encoding Different relationships carry different semantics. A `purchased` relationship differs from a `reports-to` relationship, so the model uses relation-aware attention with specialized weights for edge types. ### Scalable subgraph sampling Enterprise graphs can contain millions of nodes and billions of edges. The source experimental setup samples two hops of 15 neighbors before applying full attention within each local subgraph. **Analogy:** A GNN detective relies on hearsay beyond direct contacts, while a standard-transformer detective interviews everyone in a city equally. A Relational Graph Transformer focuses on a relevant group but can attend broadly when evidence warrants it. ## Positional encodings for relational structure Graphs have no natural linear order, so Relational Graph Transformers use four composable local encodings computed within each sampled subgraph. | Encoding | What it captures | Computation | Key benefit | | --------------- | ---------------------------------- | -------------------- | ------------------------------------- | | Hop | Graph distance | Local per subgraph | Explicit proximity signal | | Tree | Parent-child hierarchy | Local per subgraph | Distinguishes siblings from ancestors | | Message passing | Local topology and structural role | Lightweight GNN pass | Approximates node2vec cheaply | | Time | Temporal ordering | Local per subgraph | Prevents leakage and captures recency | Hop encoding identifies distance from the target node. Tree encoding represents parent-child hierarchy. Message-passing encoding refines random embeddings with a lightweight GNN to approximate node2vec-style structural features. Time encoding limits information to earlier events, preventing data leakage while capturing recency. ## Benchmark results on RelBench The source evaluates models on [RelBench](https://relbench.stanford.edu), a public benchmark for predictive tasks over relational databases. Both GNN and transformer models use two-hop sampling with 15 neighbors, a node dimension of 128, and the same local neighborhoods. The transformer uses four layers, eight attention heads, and a 512-dimensional feed-forward network. LightGBM is an additional baseline using raw entity-table features without graph feature engineering. The source reports that Graph Transformers outperform the GNN baseline by around 10% and LightGBM by more than 40% across RelBench datasets. | Model | Architecture | vs. LightGBM | vs. GNN baseline | | ---------------------------- | ------------------------------------------------ | -------------: | ---------------: | | LightGBM | Gradient-boosted trees on flat features | Baseline | \~40% below | | GNN | Message passing, two-hop sampling, dimension 128 | \~40% above | Baseline | | Relational Graph Transformer | Four layers, eight heads, dimension 128, FFN 512 | Over 40% above | \~10% above | The comparison holds sampling constant. The claimed transformer advantage comes from allowing all nodes in the sampled subgraph to attend directly rather than aggregating strictly one hop at a time. ## From benchmark to production Production systems must handle graphs with millions of nodes and billions of edges, low-latency predictions, cold-start entities, and sparse data. Schema-guided sampling focuses on the tables and relationships relevant to a task. With two-hop sampling at 15 neighbors per hop, the source estimates a bounded subgraph of at most about 240 nodes per prediction. The forward pass scales with the sampling budget rather than the full database size. Cold-start entities can still contribute their own multi-modal attributes before interaction history accumulates. The source reports these production benefits for Kumo.ai deployments: * **20x faster time-to-value** than traditional feature-engineering and model-training pipelines. * **30-50% accuracy improvements** from deeper relational context. * **95% less data-preparation effort** by operating directly on relational structure. ## Implications for foundation models ![Comparison of traditional ML pipelines, GNNs on relational data, and relational graph transformers as a foundation-model architecture.](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nvidia-sdgm.docs.buildwithfern.com/56d450974eec480f93a099a426aec3f33ca39803434424fbfea494e44c95892d/img/research/relational-graph-transformers-foundation-comparison.svg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260920T164627Z&X-Amz-Expires=604800&X-Amz-Signature=de1d2882aeb930376123b7266cee4d4c76e840ab66e7d85e114315bbc8ffec0d&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) Relational Graph Transformers provide an architectural basis for foundation models over relational data. The source highlights schema-agnostic attention, multi-modal feature encoding, and composable positional encodings as properties that support transfer across databases and tasks. The work builds on [Relational Deep Learning](https://arxiv.org/abs/2312.04615), [PyTorch Frame](https://arxiv.org/abs/2404.00776), and [RelBench](https://arxiv.org/abs/2407.20060). The source positions KumoRFM as a model built on this direction, with remaining research questions focused on larger and more diverse pretraining datasets, schemas, and models.