GPU Sizing for Docker
This page covers GPU host sizing for a Kumo Docker Compose deployment. GPU support is optional for evaluation; training and prediction jobs fall back to CPU when no GPU is present. For production GPU workloads, use the sizing tables below to select the right host before deploying.
Two numbers drive every decision:
G_max is the size Kumo reports, not the size your warehouse reports.
Kumo measures the exported Parquet it writes to blob storage, which is typically larger than the figure shown in a warehouse console - a 2-3x difference is common.
Budget 3x your warehouse estimate when planning a first deployment, and verify against the graph size shown in the Kumo UI.
How Docker deployments use GPUs
There is no pool in a Docker deployment. Kumo runs one worker on one host, with no autoscaling, no overflow pool, and no warm-cache routing. Sizing means sizing a single machine.
Concurrency is a fixed setting rather than an elastic one.
The POOL_GPU_CONCURRENCY environment variable controls how many training and prediction jobs run at once.
It ships set to 1.
Every job beyond that count queues.
Leaving POOL_GPU_CONCURRENCY at 1 on a multi-GPU host is the most common and most costly Docker misconfiguration.
In Kumo’s test data, jobs that queued behind another job took a median of 148 minutes against 32.5 minutes for jobs that ran immediately - roughly a 4.5x penalty from serialization.
The GPU Compose overlay (docker-compose.singlehost.gpu.yml) exposes all available NVIDIA devices to the pool worker container using count: all.
POOL_GPU_CONCURRENCY is the only throttle; set it to your physical GPU count.
Choosing a size
Match the GPU count to your chosen tier first. vCPU and RAM vary between instance families, so you may need a larger variant than the minimum GPU count implies.
Reference instances:
On Azure, there is no 48 GB VRAM option. Azure steps from 24 GB to 80 GB or above. Treat the choice as binary: use 24 GB unless link prediction over large target tables forces you higher.
Configuration
POOL_GPU_CONCURRENCY - set this to your physical GPU count.
This is the highest-value change available on a multi-GPU host.
It is hardcoded as "1" in docker-compose.singlehost.yml under the kumo-pool-worker service; edit that value directly in the compose file (or supply a compose override file).
Setting it in .env alone has no effect because the compose file does not use variable substitution for this key.
- Never set it above your GPU count. This value is the only safeguard preventing two training processes from contending for a single device.
DISKGRAPH_DATA_ROOT - the host path where DiskGraph stores graph indexes.
Default: /mnt/localssd.
Point this at a fast local SSD for real workloads.
Size local disk at roughly 2x G_max (Kumo requires approximately 1.2x for the index, plus room for staging tables and temporary files).
shm_size - set in docker-compose.yml on the kumo-pool-worker service, not in .env.
The blueprint ships with shm_size: 8g.
For L and XL hosts, increase it to the value shown in the sizing table above by editing the Compose file directly.
Keep ipc: host and shm_size as shipped.
Both are required for the GPU subprocess flow.
Removing either causes training failures that are difficult to diagnose.
Add memory limits to your Compose file for production use. The shipped file does not define them, so a runaway training process can exhaust the host. Set per-service limits appropriate to your available RAM.
When to move to Helm
Move to a Kubernetes deployment when any of the following becomes true:
- You need more than approximately four concurrent training jobs.
- Your graphs exceed approximately 250 GB.
- You need elasticity - a Docker host incurs full cost whether idle or busy.
- You need warm-cache reuse across many graphs.
There is no in-place migration path between the two. Plan your deployment path before Docker becomes production-critical.