> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server. # GPU Sizing for Helm > How to size GPU node pools for a Kumo Helm deployment on Kubernetes. This page sizes the GPU capacity for a Kumo Helm deployment: how many GPUs to provision, how much VRAM each node needs, and how much local SSD to attach. It covers EKS, AKS, and GKE. Two numbers drive every decision: | Input | What it means | How to find it | | ------- | --------------------------------------------------- | ----------------------------------------------------------------- | | `G_max` | The source size of your largest single graph, in GB | Sum the source table sizes of the largest graph you plan to build | | `C` | Your peak concurrent training and prediction jobs | How many models will realistically train at the same moment | `G_max` determines disk. `C` determines GPU count. They are independent, and neither one determines VRAM. `G_max` is the size Kumo reports, not the size your warehouse reports. Kumo measures the exported Parquet it writes to blob storage, which is typically larger than the figure shown in a Snowflake or Databricks console - a 2-3x difference is common. Use the graph size shown in the Kumo UI, or budget 3x your warehouse estimate when planning a first deployment. ## 1. How Kumo uses GPUs A Helm deployment runs two GPU node pools per tenant. * **Cache pool - warm storage and baseline execution.** Long-lived pods that hold built graph indexes on local NVMe SSD. Building an index is the expensive stage of the pipeline, so keeping it warm allows repeat jobs against the same graph to skip ingestion entirely. * **Ephemeral pool - overflow.** Short-lived pods that start when a job cannot run on a cache node and shut down when it finishes. They rebuild what they need on local disk, so they are slower. This pool absorbs bursts; it is not intended for steady-state work. Three behaviors govern every sizing decision: 1. **One GPU per node.** Every training and prediction pod requests exactly one GPU, and every cache node runs one job at a time. This prevents two jobs from silently contending for the same device. 2. **Peak concurrent jobs equals peak node count.** Because of the first rule, the autoscaler adds one node per concurrent job. There is no bin-packing to plan around, and a multi-GPU instance does not increase concurrency. 3. **A cache node with a busy GPU sends the next job to the ephemeral pool**, even when that node holds the warm copy of the graph. Cache node count is therefore a concurrency setting as much as a storage setting. ## 2. Choosing a size | | S | M | L | XL | | ------------------------ | -------------------- | --------------------------- | -------------- | ----------------- | | Largest graph `G_max` | ≤ 10 GB | ≤ 50 GB | ≤ 100 GB | ≤ 300 GB | | Peak concurrent jobs `C` | 2 | 4 | 8 | 16 | | Cache nodes | 1 | 2 | 4 | 8 | | Ephemeral nodes | 1 | 2 | 4 | 8 | | **Total GPUs** | **2** | **4** | **8** | **16** | | VRAM per GPU | 24 GB | 48 GB | 48 GB | 80 GB | | Local NVMe per node | ≥ 200 GB | ≥ 400 GB | ≥ 800 GB | ≥ 1750 GB | | Typical use | Pilot or single team | Most production deployments | Multiple teams | Very large graphs | Start at **M** unless you have a specific reason not to. It is the size most production deployments settle on, and it absorbs a busy multi-user workload with only occasional queueing. ## 3. Selecting an instance type The instance types below are validated for each cloud. All are single-GPU. Do not substitute a multi-GPU SKU expecting additional concurrency - Kumo uses one GPU per node regardless of how many the instance provides. | | AWS (EKS) | Azure (AKS) | Google Cloud (GKE) | | ----------------- | ----------------- | -------------------------- | -------------------- | | GPU instance | `g6e.8xlarge` | `Standard_NC40ads_H100_v5` | `g2-standard-16` | | GPU | 1x L40S, 48 GB | 1x H100 NVL, 94 GB | 1x L4, 24 GB | | vCPU and RAM | 32 vCPU, 256 GiB | 40 vCPU, 320 GiB | 16 vCPU, 64 GiB | | Local NVMe | \~450 GiB | 3576 GiB | 8 x 375 GB (3000 GB) | | `ssd_gib` value | `400` | `3576` | `3000` | | CPU pool instance | Per your baseline | `Standard_D32s_v5` | `n2-standard-32` | ### 3.1 Azure notes **There is no 48 GB option on Azure.** The NVIDIA VRAM ladder available on Azure skips it, running from T4 at 16 GB to A10 at 24 GB to A100 at 80 GB to H100 NVL at 94 GB. `Standard_NC40ads_H100_v5` is listed above because it is the current-generation single-GPU SKU with local NVMe large enough for the cache pool, not because the workload requires 94 GB of VRAM. If cost matters more than headroom, evaluate a 24 GB A10 SKU against the guidance in section 6 and confirm its local NVMe capacity meets your `G_max`. **Create GPU node pools with `--node-osdisk-type Managed`.** With `Ephemeral`, AKS allocates the local NVMe disk for the operating system, leaving no space for Kumo's graph storage. ### 3.2 Google Cloud notes **There is no 48 GB option on Google Cloud either.** The single-GPU ladder runs from T4 at 16 GB to L4 at 24 GB to A100 at 40 GB and 80 GB. `g2-standard-16` (1x L4, 24 GB) is listed above because the G2 family pairs a single GPU with attachable local NVMe at reasonable cost. That is sufficient for S and most M workloads, but 24 GB is the floor described in section 6. If you run link prediction over large tables, evaluate a single-GPU A100 SKU instead and confirm its local SSD capacity meets your `G_max`. Provision G2 node pools with `--local-nvme-ssd-block=count=8`. ### 3.3 Local NVMe preparation **This step applies to EKS only.** On Azure (AKS), cache PVCs use zonal Azure Disk (`managed-csi-xfs`), not local NVMe, so no volume group preparation is needed on AKS GPU nodes. On EKS, the local NVMe on each GPU node must be assembled into an LVM volume group named `topolvm-vg` before Kumo starts. Kumo's TopoLVM StorageClass manages logical volumes within that group but does not create the group. This is handled by a node startup script included in your deployment package. To validate volume group assembly on an EKS node: ```bash kubectl debug node/ -it --image=ubuntu -- \ chroot /host bash -c "lsblk && pvs && vgs" ``` Expect `topolvm-vg` backed by a physical volume of approximately the full local NVMe size. ## 4. Configuration Two settings must agree, or the scheduler will never place a workload. ### 4.1 Declare the tier Add the tier to your Helm values, using the `ssd_gib` value from the table in section 3: ```yaml kumoConfig: diskgraph: nodeTypes: g6e.8xlarge: { ssd_gib: 400 } # AWS # Azure: Standard_NC40ads_H100_v5: { ssd_gib: 3576 } # GCP: g2-standard-16: { ssd_gib: 3000 } ``` The key is a name only, so any string works. What matters is that `ssd_gib` matches the node labels below exactly. ### 4.2 Label the node pools | Pool | Required labels | | --------- | --------------------------------------------------------------------------------------------------------------------------- | | Cache | `kumo.ai/diskgraph-pool-type=cache` `kumo.ai/diskgraph-ssd-size=400` (must equal `ssd_gib`) `kumo.ai/diskgraph-gpu-count=1` | | Ephemeral | `kumo.ai/diskgraph-pool-type=ephemeral` | The ephemeral pool takes no size or GPU count label, but its nodes still need a GPU and sufficient fast local disk. ### 4.3 Set the node count ceiling This step is most often missed. If your autoscaler caps a node pool by total CPU rather than by node count, that cap is your real GPU ceiling: ``` cpu limit = number of nodes x vCPU per instance ``` For size M on AWS, that is `2 nodes x 32 vCPU = 64`. Increasing your intended GPU count without raising this limit changes nothing - the pool silently remains at its previous size. ## 5. Verification Run the conformance checker included in your deployment package and confirm three checks pass: * `DISKGRAPH_POOLS` - the pools exist and their labels match the declared tiers. * `GPU` - nodes advertise `nvidia.com/gpu`. The probe does not check the driver version; verify that the NVIDIA driver on each GPU node meets the minimum version requirement separately. * `CSI_NODE_COVERAGE` - the storage driver is registered on every cache node. Kumo also runs its own canary at startup, creating a test volume and GPU pod request against each declared tier. If the canary does not schedule within approximately 10 minutes, the controller reports which tier failed and what is missing. ## 6. Choosing VRAM **VRAM does not scale with graph size.** A 300 GB graph streams through the sampler in batches and is never resident in GPU memory. VRAM scales with batch size, embedding dimension, and the row count of link prediction target tables. Disk and VRAM are therefore independent choices, and a large graph does not by itself require a large GPU. What happens when VRAM is short depends on the task type: * **Node prediction, classification, regression, and forecasting.** Kumo catches the out-of-memory condition, halves the batch size, and retries. Undersized VRAM costs time, not correctness. * **Link prediction.** There is no equivalent fallback. Jobs over large target tables fail with an explicit error asking you to reduce `output_embedding_dim`. Link prediction over large tables is therefore what sets your VRAM floor. | VRAM | Guidance | | --------------- | --------------------------------------------------------------------------------------- | | 16 GB | Evaluation only | | 24 GB | Minimum for production. Suitable for S, and for M unless you run heavy link prediction. | | 48 GB | Recommended default. Handles every workload class through L. | | 80 GB and above | Only for XL, and specifically for link prediction over very large target tables. | 48 GB is directly available only on AWS. Neither Azure nor Google Cloud offers a 48 GB NVIDIA option; both step from 24 GB to 80 GB or more. On those clouds, treat the choice as binary: use 24 GB unless link prediction over large target tables forces you higher, in which case move to 80 GB or above. If you are running 24 GB GPUs and encounter link prediction memory errors, reduce `output_embedding_dim` first. That is usually sufficient and costs considerably less than changing instance families. ## Quick reference | Question | Answer | | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | How many GPUs do I need? | Your peak concurrent job count. One GPU per node, one job per GPU. | | How much VRAM per GPU? | 48 GB by default, 24 GB minimum, 80 GB only for XL link prediction. | | How much local SSD per node? | `G_max x 1.2` minimum; `G_max x 2` is comfortable. | | Does a larger graph need a larger GPU? | No. A larger graph needs more disk. Larger batches, embeddings, or link prediction tables need more VRAM. | | Will a 4-GPU instance run four jobs? | No. Kumo uses one GPU per node. Add nodes rather than GPUs per node. | | Is 48 GB available on every cloud? | No, only on AWS. Azure and Google Cloud both step from 24 GB to 80 GB or more. | | What are the most common mistakes? | Not raising the autoscaler CPU cap alongside GPU count; node labels not matching `ssd_gib` exactly; using `--node-osdisk-type Ephemeral` on AKS; assuming AKS local NVMe assembly is automated. |