GPU Compute Modes

GPU memory fractions and dynamic GPU memory fractions determine how much GPU memory each workload receives. GPU compute modes determine how the workloads sharing a GPU divide the GPU compute between them, and are set in addition to a memory fraction, whether static or dynamic. GPU compute modes are part of the CUDA-based GPU fractions implementation.

NVIDIA Run:ai offers two compute modes - time-slicing and SM-sharing. Both are set per workload, both work with every workload type, and both are available on any GPU that meets the prerequisites. Time-slicing is the default; SM-sharing must be set explicitly.

A GPU runs one compute mode at a time. The first workload placed on a GPU sets the compute mode of that GPU, workloads requesting the other mode are not placed on it, and the mode resets when the GPU empties.

How the Modes Differ

Under full contention, both modes give each workload roughly an equal share of the GPU compute. The difference is whether the workloads take turns on the GPU or run at the same time.

Both modes also redistribute compute that nobody is using. Consider three workloads sharing one GPU:

• While all three are busy, each receives roughly 33% of the GPU compute.
• If one workload goes idle, its share is split evenly among the other two, and each receives roughly 50%.
• When the idle workload becomes active again, all three return to roughly 33%.

A single active workload on the GPU can therefore use the whole GPU compute. Compute is not held back from workloads that can use it.

As a rule of thumb, use time-slicing for work that arrives in bursts and can wait for its turn. Use SM-sharing when a workload needs the GPU steadily, or when something downstream is waiting on its output.

How GPU Compute Modes Work

Time-Slicing

Time-slicing is serial. Workloads take turns on the whole GPU, and each turn gives one workload exclusive use of the GPU compute engines. When every workload uses its full turn, each receives roughly an equal share of the GPU time. Time left unused by one workload goes to the others, up to the whole GPU for a single active workload.

Only one workload runs at a time, so each workload is its own error domain. A fatal fault in one workload does not affect the others sharing the GPU.

Time-slicing is the default compute mode.

SM-Sharing

SM-sharing is parallel. The GPU streaming multiprocessors (SMs) are divided between the workloads and all of the workloads run at the same time. At any instant an SM serves one workload and no other.

Nothing is carved out in advance. SMs left idle by one workload are used by the others, up to the whole GPU for a single active workload.

SM-sharing runs the workloads through a shared context, so the workloads sharing a GPU also share an error domain. A fault that is attributed to the GPU context cannot be traced back to the process that caused it, so a fatal fault affects every workload sharing that GPU.

Prerequisites

GPU compute modes require the following on every GPU node in the cluster. GPU nodes must also run Linux with cgroup v2 mounted at /sys/fs/cgroup, and since GPU Operator 26.7.1 does not default to CUDA driver r615, set the driver version explicitly rather than relying on the default.

Setting the GPU Compute Mode

The compute mode is set per workload with the nvidia.com/container.<container-name>.gpu-compute.mode annotation, whether you submit through the UI, the CLI, the API, or directly in YAML. Set this in addition to the GPU memory fraction or dynamic GPU memory fraction request - the compute mode has no effect without a memory fraction already requested on the same container. After the workload is submitted, the compute mode is visible in the workload’s YAML tab. For a full step-by-step walkthrough of each interface, refer to Launching Workloads with GPU Compute Modes.

Annotation Reference

Substitute <container-name> with the name of the container receiving the GPU memory fraction. A GPU fraction is always allocated to a single container, and NVIDIA Run:ai currently supports one fractional container per pod. This is the same container named by gpu-fraction-container-name, or the first container in the pod when that annotation is omitted. If both are used, the container names must be identical or the request is rejected.

If the annotation is omitted, time-slicing applies.

This annotation applies the same way to supported workload types submitted via YAML and to workloads submitted externally, outside of NVIDIA Run:ai.

Enforcing a Compute Mode with a Policy

Administrators can also control the compute mode centrally with a workload policy. A policy can supply a default compute mode for workloads that do not set one, or enforce a specific mode so that users cannot change it. This is available for both native workloads and supported workload types.

Example

The following example requests an absolute amount of GPU memory and enables SM-sharing:

metadata:
  annotations:
    gpu-memory: "20000"   # MiB
    # "time-slicing" (the default, may be omitted) or "sm-sharing"
    nvidia.com/container.app.gpu-compute.mode: "sm-sharing"
spec:
  containers:
    - name: app          # the <container-name> used in the mode annotation

Deployment Considerations

• The compute mode is exclusive per GPU. A GPU running time-slicing cannot also run SM-sharing, and the reverse. The first workload placed on a GPU fixes the compute mode of that GPU. Every workload that joins afterwards must request the same mode, and the mode resets when the GPU empties.
• Mixing compute modes reduces the GPUs available to each mode. While a GPU is running one compute mode, workloads requesting the other mode cannot be placed on it. On a cluster where both modes are in regular use, a workload can wait for a GPU even though GPU memory is free elsewhere in the node pool. A workload policy that sets a default compute mode per project or department keeps a team’s workloads consistent with each other.
• Per-container compute metrics are not available under SM-sharing. Workloads using SM-sharing share a context, and compute utilization cannot be attributed to an individual container. GPU memory metrics are reported per container in both modes.

Using CLI

To view the available actions, go to the CLI v2 reference and run according to your workload.

Using API

To view the available actions, go to the API reference and run according to your workload.