> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Kumo Engine Benchmarks

> Measured context-generation, training, and batch-prediction performance, with datasets, hardware, and comparison scope.

Kumo Engine generated three-hop neighborhoods for one million entities in **19 seconds with a warm cache** on a graph containing **50 billion nodes and 175.5 billion edges**.

The experiments below measure context generation, a commerce training and prediction workflow, and graph ML framework performance. Each comparison uses its own dataset, hardware, and measurement scope.

## Context Neighborhood Generation

Context generation retrieves connected records and their raw features for each starting entity, or seed. These neighborhoods supply model inputs for graph models, transformers, and in-context learning workloads.

This experiment used a synthetic commerce graph with **50 billion nodes, 175.5 billion edges, seven tables, and nine foreign keys**. The source contained **2.8 TB of uncompressed data**, stored as **500 GiB of Parquet files**. DiskGraph's **2.6 TiB index exceeded the Kumo host's 1 TiB RAM**.

| Methods                | Hardware                                                                                |
| ---------------------- | --------------------------------------------------------------------------------------- |
| Kumo Engine and DuckDB | One AWS i4i.32xlarge: 128 vCPUs, 1 TiB RAM, and eight striped NVMe disks totaling 30 TB |
| Polars GPU             | One AWS g6e.48xlarge: 192 vCPUs and 1,536 GiB RAM                                       |

The methods used different data representations:

* **Kumo Engine:** Reads the DiskGraph index from local NVMe storage.
* **Prepared DuckDB:** Reads native on-disk tables and key-and-time-sorted lookup tables prepared from Parquet.
* **DuckDB on Parquet:** Reads source Parquet files directly.
* **Polars GPU:** Queries sorted Parquet files with GPU execution.

### Neighborhood Retrieval Time

The following timings cover **one million seeds** and exclude preparation and model training. Kumo Engine and prepared DuckDB used warm caches by repeating the same seeds a second time. Durations are rounded.

| Depth  | Nodes per Seed | Kumo Engine, Warm Cache | Prepared DuckDB, Warm Cache | DuckDB on Parquet | Polars GPU, Sorted Parquet |
| ------ | -------------- | ----------------------- | --------------------------- | ----------------- | -------------------------- |
| 2 hops | 76             | **5 seconds**           | 4.0 minutes                 | 10.5 minutes      | 1.8 minutes                |
| 3 hops | 577            | **19 seconds**          | 5.3 minutes                 | 17.5 minutes      | 5.1 minutes                |
| 4 hops | 1,049          | **39 seconds**          | 6.5 minutes                 | 47 minutes        | 6.2 minutes                |

Across these depths, Kumo Engine was approximately **10 to 48 times faster than prepared DuckDB** and **9.5 to 22 times faster than Polars GPU**, using the rounded timings. The Polars comparison used different hardware, as shown above.

![Context-generation timings for one million seeds at two to four hops. Kumo Engine and DuckDB used an i4i.32xlarge; Polars GPU used a g6e.48xlarge.](/sdgm/_fern-img/3ed7437d2f236c3dbcedd25eab6a5a0b2ea7848d6dfb316bd2b7333e2836e429.webp)

DiskGraph and prepared DuckDB returned identical neighborhoods for every seed compared.

### One-Time Graph Preparation

These retrieval timings exclude the initial preparation step:

| Preparation                       | Time         |
| --------------------------------- | ------------ |
| DiskGraph index build             | 49.8 minutes |
| DuckDB native load and sort       | 40.1 minutes |
| DuckDB on Parquet                 | None         |
| Polars sorted Parquet preparation | 44.3 minutes |

DiskGraph took longer to prepare than the tested DuckDB and Polars representations, then retrieved neighborhoods faster. Repeated sampling can amortize that preparation cost.

## Commerce Training and Prediction Workflow

This experiment measured Kumo Engine training and batch prediction on the **50-billion-node synthetic commerce graph**, using one AWS **g6e.48xlarge**.

### Training Pipeline

Kumo Engine trained a six-hop model in approximately **5.2 hours**, including graph ingestion and training-table generation:

| Stage                          | Elapsed Time                |
| ------------------------------ | --------------------------- |
| Graph ingestion into DiskGraph | 61 minutes                  |
| Training-table generation      | 1.7 hours                   |
| Training                       | 2.4 hours                   |
| **Reported total**             | **Approximately 5.2 hours** |

Training used **1,000 steps per epoch** and a **batch size of 1,024**. The 61-minute ingestion measurement belongs to this training experiment; the 49.8-minute index build above belongs to the separate context-generation experiment.

### Reuse Across Tasks

Kumo Engine built the graph index once and reused it across three tasks. Each task added its own training-table generation and model configuration without rebuilding the index.

### Batch Prediction

The commerce graph included **1 billion unique customers** among its 50 billion nodes. Batch prediction ran on a subset of **5 million customers** using one **g6e.48xlarge**.

| Finding               | Result                                                                                                           |
| --------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Batch prediction time | Approximately **16 minutes** for **5 million customers** in the best tested configuration                        |
| Throughput bottleneck | Data loading from SSD and RAM                                                                                    |
| Worker configuration  | The default selected eight dataloader workers regardless of GPU count, limiting throughput with four GPUs active |

## Graph ML Framework Comparison

These experiments used a citation graph with **111 million nodes**, **1.616 billion citation edges** (3.23 billion including reverse edges), and **128-dimensional node features**. The prediction task had **172 classes** and **1.55 million labeled nodes** across the official training, validation, and test splits.

### PyG on Matched Hardware

Kumo Engine and an in-memory PyG implementation used the same model modules and matched training settings. Both ran for **15 epochs** on one **g5.16xlarge with one A10G GPU, 256 GiB RAM, and local NVMe**.

Kumo Engine completed training and validation in **41 minutes 14 seconds**, compared with **64 minutes 13 seconds** for PyG, a **36% reduction**. Its observed training-container memory was also lower:

| Measurement                        | Kumo Engine               | PyG, Matched Model           |
| ---------------------------------- | ------------------------- | ---------------------------- |
| Graph construction                 | **3 minutes 26 seconds**  | 4 minutes 1 second           |
| Training and validation, 15 epochs | **41 minutes 14 seconds** | 64 minutes 13 seconds        |
| Full workflow                      | **55 minutes 18 seconds** | 69 minutes 20 seconds        |
| Observed training-container memory | **17 to 37 GiB**          | Approximately 166 to 167 GiB |
| Test accuracy                      | 63.26%                    | 63.93%                       |

Kumo Engine's full workflow included ingestion, training, evaluation, and model publication. Its test accuracy was **0.67 percentage points lower** in this run, alongside the lower training time and observed memory use.

### GraphStorm on Larger Hardware

[GraphStorm](https://aws.amazon.com/blogs/machine-learning/faster-distributed-graph-neural-network-training-with-graphstorm-v0-4/) is an AWS open-source framework for graph neural network training and deployment. The configurations below stored graph data in RAM and used the same 111-million-node citation graph.

Graph construction ran on a separate **r5.24xlarge with 768 GiB RAM**. Distributed training used **four g5.48xlarge instances**; the single-node training configuration used **one g5.16xlarge**.

| Configuration          | Graph Construction            | Peak Construction RAM | Training, 15 Epochs           |
| ---------------------- | ----------------------------- | --------------------- | ----------------------------- |
| Distributed GraphStorm | 3 hours 42 minutes 33 seconds | 679 GB                | 14 minutes                    |
| Single-node GraphStorm | 51 minutes 16 seconds         | 415 GB                | Failed before the first epoch |

Distributed GraphStorm completed training faster than Kumo Engine, using more training hardware. Its graph-construction stage took longer than Kumo Engine's full 55-minute workflow. The single-node GraphStorm run failed while loading the graph and features into RAM.

### Additional Storage Examples

Separate workload examples illustrate the storage configurations used for different datasets:

![Separate dataset examples: GraphStorm with 230 million rows and 768 GiB or more RAM; Kumo Engine with 10 billion rows, 128 GB RAM, and 4 TB SSD. These examples do not establish a relative capacity limit.](/sdgm/_fern-img/59fb76ae509b446e4f29e9f137e20602099d6965a9e5bec5f8b9ba0aa7f06ccb.webp)

| Metric                | Kumo Engine              | GraphStorm                                      |
| --------------------- | ------------------------ | ----------------------------------------------- |
| Dataset size          | **10 billion rows**      | 230 million rows                                |
| Relative dataset size | **Approximately 43×**    | 1×                                              |
| Storage model         | NVMe and RAM             | In-memory graph                                 |
| Machine configuration | **128 GB RAM, 4 TB SSD** | 768 GiB or more RAM (r5.24xlarge or equivalent) |

These examples use different datasets and do not establish a relative capacity limit. They are separate from the commerce and citation-graph experiments above.

## Comparison with a DuckDB Feature-Engineering Pipeline

This experiment compares two complete modeling workflows: Kumo Engine learning from connected records, and a model trained on features engineered with DuckDB. Both pipelines ran from raw Parquet to a trained model on the same **50-billion-node synthetic commerce dataset**, using one AWS **g6e.48xlarge** per pipeline.

The DuckDB pipeline constructed model inputs with hand-written aggregates and multi-table joins. Features included column means, standard deviations, modes, and multi-hop signals such as co-purchase, co-view, and similar-customer features. Kumo Engine retrieved connected records and raw features for its graph model.

The DuckDB approach required task-specific labels, temporal windows, candidate sets, and feature logic.

| Measurement                  | Kumo Engine   | DuckDB Pipeline |
| ---------------------------- | ------------- | --------------- |
| Raw Parquet to trained model | **5.2 hours** | 9.9 hours       |
| Implementation lines of code | **40**        | 1,243           |

The code counts cover features, candidate generation, labels, and evaluation in these implementations.

### Instance Compute Cost

Because both pipelines used the same instance type, their elapsed times imply approximately **1.9 times the instance compute cost for the DuckDB pipeline**, at the same hourly rate. This comparison covers instance compute only.

## Plan Your Evaluation

Use a representative task and dataset, then:

1. Measure source ingestion and index preparation separately from repeated sampling, training, and prediction.
2. Record hardware, software versions, sampling settings, model configuration, and the full training budget.
3. Compare runtime and memory alongside held-out prediction quality, using the same evaluation population and metrics.

Use the [Model Plan Guide](/kumo-engine/1.0.0/user-code-guide/train-a-model/model-plan-guide) to configure sampling and training. For the complete workflow, start with the [Kumo Engine Overview](/kumo-engine/1.0.0/get-started-with-kumo-engine/overview).