> This page is for Kumo Engine.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Infrastructure Scaling

> Measured workflow and stage timings as table count, relationship layout, row volume, and feature width grow on one g6e.8xlarge.

We benchmarked Kumo Engine 1.0.0 on one **AWS g6e.8xlarge with a single NVIDIA
L40S GPU** to understand how database structure and size affect a complete
training and prediction workflow. We measured source staging, graph creation,
training-table generation, fit, and prediction while varying table count,
relationship layout, row volume, and feature width.

Adding tables increased runtime across the measured schema families, but the
relationships between those tables changed how much work followed. At 64 tables,
the chain took **335.2 seconds across stages**, compared with **477.4 seconds for
the star** and **498.9 seconds for the mapping schema**. Most of that difference
was in fit. The examples below show how the same five tables can form each layout.

![Chain: T4 links through T3, T2, and T1 to prediction entity T0.](/sdgm/_fern-files/nvidia-sdgm.docs.buildwithfern.com/664cac720244a29c7cb7180437704737e6c5efaee6e9c549d44dd78e551d7a6a/_dot_dot_/images/benchmarks/infrastructure-scaling/schema-chain.svg)

![Star: four child tables link directly to prediction entity T0.](/sdgm/_fern-files/nvidia-sdgm.docs.buildwithfern.com/2d38491505dc2bd7f83c71604aa25624859b47a200bcc332dfab16f42fed08a4/_dot_dot_/images/benchmarks/infrastructure-scaling/schema-star.svg)

![Tree: T1 and T2 link to prediction entity T0, while T3 and T4 link to T1.](/sdgm/_fern-files/nvidia-sdgm.docs.buildwithfern.com/a34d0720991cc38f8afc3b76284162940e9383237baec59fd97b0e1ef834c441/_dot_dot_/images/benchmarks/infrastructure-scaling/schema-tree.svg)

![Mapping: T2, T3, and T4 each link to prediction entity T0 and second parent T1.](/sdgm/_fern-files/nvidia-sdgm.docs.buildwithfern.com/5f8fcb00097429a9b6a1a3ff8e5e610a8e09ce0d5d105415101223febcde5ea1/_dot_dot_/images/benchmarks/infrastructure-scaling/schema-mapping.svg)

![Cycle: a chain of five tables closes with a relationship from T0 back to T4.](/sdgm/_fern-files/nvidia-sdgm.docs.buildwithfern.com/0436c05ecd9914be38edcedf4c1025c32e56b36b4b503460a8343179ef47a574/_dot_dot_/images/benchmarks/infrastructure-scaling/schema-cycle.svg)

*These five-table examples illustrate the relationship layouts. T0, shaded green,
is the prediction entity. Each arrow points from a child's foreign key to its
parent's primary key.*

Row growth and feature growth affected different parts of the workflow. In the
three-table row sweep, increasing the mapping table from 10 million to 100 million
rows raised graph-build time from **29.7 to 83.0 seconds**, while fit stayed near
**116 to 118 seconds**. At a fixed 10 million mapping rows, widening that table
from 2 to 500 columns increased fit from **75.7 to 923.1 seconds**.

For users planning work on a large database, these results suggest three priorities:

* **Pilot the actual schema and prediction target.** Preserve important
  relationships and mapping tables. Table count alone did not rank runtime.
* **Plan preparation and model execution separately.** More rows can add
  substantial staging and graph-build work even when the training budget stays
  fixed. Include case overhead in job-duration estimates.
* **Review feature width and measure resource use.** Wider tables increased both
  fit and prediction time. Evaluate which features the task needs, and measure
  system RAM and local scratch alongside compressed source bytes before sizing
  the full graph.

Contents

* [Timing measures and stages](#timing-measures-and-stages)
* [Schema and table-count scaling](#schema-and-table-count-scaling)
* [Row-volume scaling](#row-volume-scaling)
* [Feature-width scaling](#feature-width-scaling)
* [Memory and storage planning](#memory-and-storage-planning)
* [Methodology](#methodology)
* [Next steps and limits](#next-steps-and-limits)
* [Download measurements](#download-measurements)

## Timing measures and stages

All results on this page use g6e.8xlarge measurements. Main tables and plots show
the first measured run of each workload. **Case wall** captures the elapsed
workflow; **stage sum** adds its five timed operations.

| Stage or measure | Timed operation                                                                                                                               |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| Case wall        | Elapsed time from the runner's recorded start to completion, including setup, all five stages, validation, and local model/prediction copying |
| Stage sum        | Sum of source staging, graph build, training-table creation, fit, and prediction                                                              |
| Source staging   | Read source manifests and download source Parquet from S3 to local storage                                                                    |
| Graph build      | Create the graph snapshot and its indexes from the staged source                                                                              |
| Training table   | Generate the query's training table and split                                                                                                 |
| Fit              | Execute the fit call, including setup, sampling, training, and evaluation                                                                     |
| Predict          | Create the prediction-table object and predict for 8,192 entities                                                                             |

![Case wall time split into the five measured stages and other case overhead, comparing five 64-table schemas and five three-table mapping-row volumes.](/sdgm/_fern-files/nvidia-sdgm.docs.buildwithfern.com/9fc5294c5642c8f4ba592d68c7e13c296eecce52a438541bca0b66102514831f/_dot_dot_/images/benchmarks/infrastructure-scaling/case-wall-time-by-stage.svg)

*Each complete bar is case wall time. The pale hatched segment is case overhead
outside the five stages.*

The 64-table chain took **371.2 seconds of case wall time**, including
**335.2 seconds across stages** and **36.1 seconds of other case overhead**.
The mapping schema took 535.3 seconds of case wall time and 498.9 seconds across stages.

### Stage timings at 64 tables

| Schema  | Case wall s | Stage sum s | Staging s | Graph s | Training table s | Fit s | Predict s |
| ------- | ----------- | ----------- | --------- | ------- | ---------------- | ----- | --------- |
| Chain   | 371.2       | 335.2       | 5.8       | 65.3    | 9.9              | 198.1 | 56.1      |
| Star    | 514.1       | 477.4       | 6.1       | 65.8    | 10.0             | 328.6 | 66.9      |
| Tree    | 377.7       | 341.8       | 5.7       | 65.3    | 9.8              | 204.7 | 56.3      |
| Mapping | 535.3       | 498.9       | 6.1       | 68.3    | 11.3             | 341.7 | 71.4      |
| Cycle   | 373.3       | 336.9       | 5.5       | 65.2    | 9.8              | 200.7 | 55.7      |

Times are seconds, rounded independently to one decimal place. Displayed stages
can therefore differ slightly from the displayed sum. Unrounded values are
available in the [raw run table and individual stage records](#download-measurements).

## Schema and table-count scaling

Each table in this experiment contains 100,000 rows and 32 columns. Adding tables
therefore increases the source volume as well as changing the graph. The layouts
shown above connect those tables in different ways.

![Total stage time and fit time versus table count for chain, star, tree, mapping, and cycle schemas.](/sdgm/_fern-files/nvidia-sdgm.docs.buildwithfern.com/72c2c9a01db0e42009202b6450823d342333769ce763776a17e1b3f58fea4077/_dot_dot_/images/benchmarks/infrastructure-scaling/table-count-and-schema.svg)

*Lines connect measured workloads. The left panel shows stage sum; the right
shows fit. Both use the same table-count axis.*

The table reports stage sums for every measured schema and table count. The source
column gives the chain's total compressed Parquet size as a reference. Other
schemas have similar source sizes; their exact values are in the downloads.

| Tables | Chain source GB | Chain s | Star s  | Tree s  | Mapping s | Cycle s |
| ------ | --------------- | ------- | ------- | ------- | --------- | ------- |
| 2      | 0.016           | 25.0    | Not run | Not run | Not run   | 25.6    |
| 4      | 0.032           | 35.9    | 38.3    | 37.9    | 39.3      | 38.0    |
| 8      | 0.065           | 55.6    | 66.6    | 62.4    | 68.7      | 58.4    |
| 16     | 0.131           | 95.0    | 122.1   | 102.1   | 127.7     | 97.3    |
| 32     | 0.263           | 172.0   | 236.3   | 178.4   | 250.1     | 178.5   |
| 64     | 0.527           | 335.2   | 477.4   | 341.8   | 498.9     | 336.9   |

The two-table star and tree are equivalent to the two-table chain and were not
run separately. The mapping experiment starts at four tables.

### Chain

Each successive table links to the preceding table. Increasing from 2 to 64 tables
raised stage sum from **25.0 to 335.2 seconds (13.4 times)**, while compressed
source size grew 33.5 times. From 32 to 64 tables, stage sum grew from 172.0 to
335.2 seconds, or **1.95 times**. Fit accounted for 198.1 seconds at 64 tables.

### Star

Every child table links to the central prediction entity. At 64 tables, star and
chain use the same source tables and **0.527 GB of Parquet**, each with 63 declared
relationships. Star's stage sum was **42.4% longer** than chain's, while graph
build was nearly unchanged at **65.8 versus 65.3 seconds**.

Fit increased from **198.1 seconds for chain to 328.6 seconds for star**, accounting
for about **92% of the additional stage time**. Within the star family, moving
from 32 to 64 tables increased stage sum from 236.3 to 477.4 seconds, or **2.02 times**.

### Tree

Tables form a binary branching hierarchy. Stage sum grew from **37.9 seconds at
four tables to 341.8 seconds at 64 tables**. The 64-table tree uses the same source
tables and relationship count as the chain and star. Its **204.7-second fit** and
341.8-second stage sum stayed close to the chain's 198.1 and 335.2 seconds.

### Mapping

Each mapping table links to two entity/metadata parents. Stage sum grew from
**39.3 seconds at four tables to 498.9 seconds at 64 tables**. At 64 tables, the
schema has **124 declared relationships**, compared with 63 for the chain, and
uses different table roles.

Its stage sum was **48.8% longer** than the chain's, with fit increasing from
**198.1 to 341.7 seconds**. From 32 to 64 tables, mapping stage sum grew from
250.1 to 498.9 seconds, or **1.99 times**.

### Cycle

A closing relationship connects the ends of a chain. Stage sum grew from
**25.6 seconds at two tables to 336.9 seconds at 64 tables**. At 64 tables, graph
build took **65.2 seconds** and fit took **200.7 seconds**, close to the chain's
65.3 and 198.1 seconds in this experiment.

Together, the matched chain, star, and tree results show why relationship layout
belongs in a sizing pilot. Their source bytes and relationship counts match, yet
their fit times differ. That is consistent with different sampling and model
execution work around the prediction entity; the stage timings do not isolate
the individual operations responsible inside fit.

## Row-volume scaling

This experiment holds the schema at three tables. Two parent tables each contain
1 million rows and 32 columns. The mapping table also has 32 columns, and only its
row count changes. Even the smallest case therefore contains more than 2 million
total source rows.

![Stage times versus mapping-table row count and column count, with total compressed source GB annotated at each measured point.](/sdgm/_fern-files/nvidia-sdgm.docs.buildwithfern.com/c9267c17b51aa161a7e82f6bbf9392ef61d883e7f9a33dcca16ebe3d5137fc9f/_dot_dot_/images/benchmarks/infrastructure-scaling/data-volume-and-feature-width.svg)

*The left panel varies mapping rows with 32 columns. The right varies mapping
columns with 10 million rows. Both contain three tables, and the y-axis scales differ.*

| Mapping rows | Source GB | Case wall s | Stage sum s | Staging s | Graph s | Training table s | Fit s | Predict s |
| ------------ | --------- | ----------- | ----------- | --------- | ------- | ---------------- | ----- | --------- |
| 1,000        | 0.167     | 158.2       | 119.6       | 1.2       | 16.2    | 3.4              | 80.1  | 18.8      |
| 100,000      | 0.175     | 166.4       | 123.5       | 6.0       | 17.0    | 3.5              | 77.8  | 19.2      |
| 1,000,000    | 0.254     | 174.1       | 158.6       | 2.1       | 24.0    | 5.0              | 100.2 | 27.2      |
| 10,000,000   | 1.034     | 200.0       | 188.4       | 5.2       | 29.7    | 6.0              | 116.1 | 31.4      |
| 100,000,000  | 8.843     | 287.3       | 271.5       | 32.7      | 83.0    | 6.1              | 118.2 | 31.5      |

From **1 million to 100 million mapping rows**, total source size increased from
0.254 to 8.843 GB, about **34.9 times**. Stage sum increased **1.71 times**, from
158.6 to 271.5 seconds. Graph build rose from 24.0 to 83.0 seconds, while fit
rose from 100.2 to 118.2 seconds.

The upper interval shows where the added work went. From **10 million to
100 million mapping rows**, source bytes increased **8.55 times**. Staging grew
from **5.2 to 32.7 seconds** and graph build from **29.7 to 83.0 seconds**. Fit
changed from **116.1 to 118.2 seconds**, and prediction from **31.4 to 31.5 seconds**,
under the same training budget and prediction cohort.

## Feature-width scaling

The mapping table stays at 10 million rows while its total column count grows.
Both parent tables keep their rows and 32-column widths. Width 2 means a mapping
table with only two foreign-key columns. Wider mapping tables add a primary key
and feature columns, so the first transition includes a structural change.

| Mapping columns | Source GB | Case wall s | Stage sum s | Staging s | Graph s | Training table s | Fit s | Predict s |
| --------------- | --------- | ----------- | ----------- | --------- | ------- | ---------------- | ----- | --------- |
| 2               | 0.233     | 139.5       | 128.9       | 2.2       | 28.6    | 3.3              | 75.7  | 19.0      |
| 8               | 0.383     | 151.6       | 137.8       | 2.6       | 33.0    | 4.4              | 76.5  | 21.3      |
| 128             | 3.639     | 453.8       | 436.4       | 14.6      | 52.3    | 13.5             | 280.3 | 75.6      |
| 500             | 13.730    | 1452.6      | 1425.7      | 52.7      | 175.1   | 36.7             | 923.1 | 238.1     |

From **2 to 500 mapping columns**, stage sum increased **11.1 times**, from
128.9 to 1,425.7 seconds. Fit increased **12.2 times**, from 75.7 to 923.1 seconds,
and prediction increased **12.5 times**, from 19.0 to 238.1 seconds. The widest
case took **1,452.6 seconds (24.2 minutes)** of case wall time.

Feature growth increased model execution time substantially even with the same
optimizer-step and prediction-count limits. Fit accounted for about **65%** of
the widest case's stage sum. Reviewing relevant features can therefore matter
alongside planning for more rows. Evaluate held-out quality before excluding
columns or changing sampling settings.

## Memory and storage planning

Two additional three-table workloads illustrate why source bytes alone do not
rank resource demand. A compact billion-row mapping table used more sampled
system memory than a substantially larger, wider source:

| Three-table workload | Mapping rows / columns | Source GB | Case wall s | Stage sum s | System memory GiB | Scratch growth GiB |
| -------------------- | ---------------------- | --------- | ----------- | ----------- | ----------------- | ------------------ |
| Compact mapping      | 1,000,000,000 / 2      | 6.816     | 317.4       | 304.7       | 116.69            | 63.37              |
| Wide mapping         | 38,999,332 / 500       | 53.062    | 1970.8      | 1939.9      | 66.86             | 218.30             |

These are separate workloads. The compact mapping table has two columns and the
wide mapping table has 500. System memory is the sampled physical memory in use;
scratch growth is the increase in used space on the scratch filesystem.

Use the [GPU sizing guide](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/gpu-sizing) when provisioning: allow at least **4 times
source size in free local SSD scratch**, or **8 times for graphs with many edges**.
Recommended RAM meets both **4 GiB per vCPU** and **1.5 times source size**. Allow
additional space for staged source files and retained snapshots or outputs on the
same filesystem, then verify actual use with a representative pilot.

## Methodology

The methodology focused solely on infrastructure benchmarking: elapsed time,
system memory, and local scratch use. Predictive accuracy was outside the evaluation.

Hardware capacities are from [AWS's G6e specifications](https://aws.amazon.com/ec2/instance-types/g6e/). Each run used a
fresh graph scratch directory, and filesystem caches were dropped before graph
creation. The loader settings below are benchmark controls rather than
recommended defaults for every workload.

| Setting                     | Value                                                                        |
| --------------------------- | ---------------------------------------------------------------------------- |
| Engine and measurement date | Kumo Engine 1.0.0; November 4, 2026 (UTC)                                    |
| Compute                     | g6e.8xlarge; 32 vCPUs; 256 GiB system RAM                                    |
| Accelerator                 | One NVIDIA L40S; 48 GB nominal GPU memory                                    |
| Working storage             | Local NVMe instance store; 900 GB nominal capacity                           |
| Source data                 | Synthetic relational Parquet with numeric, categorical, and temporal columns |
| Prediction task             | Binary classification on the designated entity table                         |
| Training budget             | 200 optimizer steps: two epochs capped at 100 steps each; batch size 256     |
| Model and sampling          | GNN with 64 channels; two sampling hops, with defaults of 10 and 5 neighbors |
| Evaluation                  | Random 80/10/10 split; validation and test capped at 20 steps per pass       |
| Prediction cohort           | 8,192 entities per run                                                       |
| Data loader                 | Eight workers, standalone mode, two Rayon threads                            |

| Experiment             | What changes                        | What stays fixed                                                                               |
| ---------------------- | ----------------------------------- | ---------------------------------------------------------------------------------------------- |
| Schema and table count | Table count and relationship layout | 100,000 rows per table; 32 columns per table                                                   |
| Row volume             | Mapping-table rows                  | Three tables; 32 columns each; two parent tables of 1 million rows each                        |
| Feature width          | Mapping-table columns               | Three tables; 10 million mapping rows; two parent tables of 1 million rows and 32 columns each |

Source sizes sum all tables' compressed Parquet bytes and use decimal GB
(1 GB = 1 billion bytes). Memory and scratch use GiB (1 GiB = 1,073,741,824 bytes).

### Repeatability

Two anchor workloads were run three times:

| Case            | Metric    | Runs | Median s | Observed range s |
| --------------- | --------- | ---- | -------- | ---------------- |
| `star-16`       | Stage sum | 3    | 122.07   | 122.05 to 122.78 |
| `star-16`       | Case wall | 3    | 139.19   | 138.90 to 141.00 |
| `rows-10000000` | Stage sum | 3    | 189.20   | 188.39 to 189.49 |
| `rows-10000000` | Case wall | 3    | 202.19   | 200.02 to 202.99 |

These ranges describe the anchors. Other workloads have one baseline run, so
small differences should not be treated as established rankings. Main tables
and plots consistently retain the first run rather than mixing it with repeat medians.

## Next steps and limits

The table-count experiment grows total rows and source bytes as it adds tables.
The controlled row-volume sweep exists only at three tables. There is no measured
volume sweep within the 4-, 8-, 16-, 32-, or 64-table schemas. The 64-table graphs
contain about **0.527 GB of source Parquet**; these runs do not establish capacity
or runtime for large 64-table databases.

The row-volume experiment fixes optimizer steps and prediction rows. Larger
training budgets, model plans, or prediction cohorts need separate measurements.
The memory sampler ran every two seconds, so shorter peaks may be missed.
Observed resource peaks are not minimum safe provisioned capacities.

For your next evaluation, preserve the intended query, relationship structure,
feature widths, and representative key distributions. Measure progressively larger
row volumes within that schema. Record case wall time, individual stages, system
RAM, GPU memory, and scratch use, then choose capacity and training settings from
those measurements. The [Model Plan Guide](https://docs.nvidia.com/sdgm/kumo-engine/user-code-guide/train-a-model/model-plan-guide)
explains the settings to record and control.

## Download measurements

The tables above cover 27 topology workloads, five row-volume workloads, and four
width workloads. The downloads include **all 48 measured g6e.8xlarge baseline runs
across 44 distinct workloads**, including encoding and stress experiments and four
additional anchor repetitions.

Raw run table (CSV)

Individual stage records (CSV)

Both tables with a column guide (ZIP)