GPU Sizing for Helm

This page sizes the GPU capacity for a Kumo Helm deployment: how many GPUs to provision, how much VRAM each node needs, and how much local SSD to attach. It covers EKS, AKS, and GKE.

Two numbers drive every decision:

G_max determines disk. C determines GPU count. They are independent, and neither one determines VRAM.

1. How Kumo uses GPUs

A Helm deployment runs two GPU node pools per tenant.

• Cache pool - warm storage and baseline execution. Long-lived pods that hold built graph indexes on local NVMe SSD. Building an index is the expensive stage of the pipeline, so keeping it warm allows repeat jobs against the same graph to skip ingestion entirely.

• Ephemeral pool - overflow. Short-lived pods that start when a job cannot run on a cache node and shut down when it finishes. They rebuild what they need on local disk, so they are slower. This pool absorbs bursts; it is not intended for steady-state work.

Three behaviors govern every sizing decision:

1. One GPU per node. Every training and prediction pod requests exactly one GPU, and every cache node runs one job at a time. This prevents two jobs from silently contending for the same device.

2. Peak concurrent jobs equals peak node count. Because of the first rule, the autoscaler adds one node per concurrent job. There is no bin-packing to plan around, and a multi-GPU instance does not increase concurrency.

3. A cache node with a busy GPU sends the next job to the ephemeral pool, even when that node holds the warm copy of the graph. Cache node count is therefore a concurrency setting as much as a storage setting.

2. Choosing a size

Start at M unless you have a specific reason not to. It is the size most production deployments settle on, and it absorbs a busy multi-user workload with only occasional queueing.

3. Selecting an instance type

The instance types below are validated for each cloud. All are single-GPU. Do not substitute a multi-GPU SKU expecting additional concurrency - Kumo uses one GPU per node regardless of how many the instance provides.

3.1 Azure notes

There is no 48 GB option on Azure. The NVIDIA VRAM ladder available on Azure skips it, running from T4 at 16 GB to A10 at 24 GB to A100 at 80 GB to H100 NVL at 94 GB. Standard_NC40ads_H100_v5 is listed above because it is the current-generation single-GPU SKU with local NVMe large enough for the cache pool, not because the workload requires 94 GB of VRAM. If cost matters more than headroom, evaluate a 24 GB A10 SKU against the guidance in section 6 and confirm its local NVMe capacity meets your G_max.

Create GPU node pools with --node-osdisk-type Managed. With Ephemeral, AKS allocates the local NVMe disk for the operating system, leaving no space for Kumo’s graph storage.

3.2 Google Cloud notes

There is no 48 GB option on Google Cloud either. The single-GPU ladder runs from T4 at 16 GB to L4 at 24 GB to A100 at 40 GB and 80 GB. g2-standard-16 (1x L4, 24 GB) is listed above because the G2 family pairs a single GPU with attachable local NVMe at reasonable cost. That is sufficient for S and most M workloads, but 24 GB is the floor described in section 6. If you run link prediction over large tables, evaluate a single-GPU A100 SKU instead and confirm its local SSD capacity meets your G_max.

Provision G2 node pools with --local-nvme-ssd-block=count=8.

3.3 Local NVMe preparation

This step applies to EKS only. On Azure (AKS), cache PVCs use zonal Azure Disk (managed-csi-xfs), not local NVMe, so no volume group preparation is needed on AKS GPU nodes.

On EKS, the local NVMe on each GPU node must be assembled into an LVM volume group named topolvm-vg before Kumo starts. Kumo’s TopoLVM StorageClass manages logical volumes within that group but does not create the group. This is handled by a node startup script included in your deployment package.

To validate volume group assembly on an EKS node:

kubectl debug node/<node> -it --image=ubuntu -- \
  chroot /host bash -c "lsblk && pvs && vgs"

Expect topolvm-vg backed by a physical volume of approximately the full local NVMe size.

4. Configuration

Two settings must agree, or the scheduler will never place a workload.

4.1 Declare the tier

Add the tier to your Helm values, using the ssd_gib value from the table in section 3:

kumoConfig:
  diskgraph:
    nodeTypes:
      g6e.8xlarge: { ssd_gib: 400 }    # AWS
      # Azure:  Standard_NC40ads_H100_v5: { ssd_gib: 3576 }
      # GCP:    g2-standard-16:           { ssd_gib: 3000 }

The key is a name only, so any string works. What matters is that ssd_gib matches the node labels below exactly.

4.2 Label the node pools

The ephemeral pool takes no size or GPU count label, but its nodes still need a GPU and sufficient fast local disk.

4.3 Set the node count ceiling

cpu limit = number of nodes x vCPU per instance

5. Verification

Run the conformance checker included in your deployment package and confirm three checks pass:

• DISKGRAPH_POOLS - the pools exist and their labels match the declared tiers.
• GPU - nodes advertise nvidia.com/gpu. The probe does not check the driver version; verify that the NVIDIA driver on each GPU node meets the minimum version requirement separately.
• CSI_NODE_COVERAGE - the storage driver is registered on every cache node.

Kumo also runs its own canary at startup, creating a test volume and GPU pod request against each declared tier. If the canary does not schedule within approximately 10 minutes, the controller reports which tier failed and what is missing.

6. Choosing VRAM

VRAM does not scale with graph size. A 300 GB graph streams through the sampler in batches and is never resident in GPU memory. VRAM scales with batch size, embedding dimension, and the row count of link prediction target tables. Disk and VRAM are therefore independent choices, and a large graph does not by itself require a large GPU.

What happens when VRAM is short depends on the task type:

• Node prediction, classification, regression, and forecasting. Kumo catches the out-of-memory condition, halves the batch size, and retries. Undersized VRAM costs time, not correctness.
• Link prediction. There is no equivalent fallback. Jobs over large target tables fail with an explicit error asking you to reduce output_embedding_dim.

Link prediction over large tables is therefore what sets your VRAM floor.

If you are running 24 GB GPUs and encounter link prediction memory errors, reduce output_embedding_dim first. That is usually sufficient and costs considerably less than changing instance families.

Quick reference