Topology-Aware Scheduling

Topology-aware scheduling places related Pods in a hardware domain with the network bandwidth they require. On an NVLink-optimized cluster, NVCF can place a multi-node function inside one GPU clique instead of spreading its Pods across slower links.

The scheduling path has four layers:

1. The NVIDIA GPU DRA driver labels nodes with nvidia.com/gpu.clique, identifying a particular NVLink partition.
2. KAI Scheduler reads a cluster-scoped Topology and places a gang in the requested domain.
3. Grove maps its workload hierarchy to the KAI topology through a ClusterTopologyBinding.
4. Dynamo uses Grove for placement of frontend, prefill, decode, and other inference services.

See the NVIDIA GPU DRA ComputeDomain guide, KAI topology guide, Grove topology guide, and Dynamo topology guide for component-level details.

Helm functions can also use the legacy dra.nvcf.nvidia.io partition annotation. That path uses Kubernetes Pod affinity and is best-effort without KAI Scheduler or Grove topology-aware scheduling. Use the KAI or Grove mechanisms on this page when clique placement must be coordinated for the complete workload.

Prerequisites

• Install the NVIDIA GPU DRA driver.
• Request a full node of GPUs for each GPU-enabled Pod in your Helm chart. The GPU DRA driver currently supports one GPU-enabled Pod per node in this mode.

See NVLink-optimized clusters for cluster registration and GPU request requirements.

Use KAI Scheduler only

Enable KAI Scheduler in the compute plane stack during installation:

1
addons:
2
  topologyAwareScheduling:
3
    enabled: true
4
  kaiScheduler:
5
    enabled: true
6
  groveOperator:
7
    enabled: false
8
  dynamoOperator:
9
    enabled: false

Topology levels are ordered from the widest domain to the narrowest domain. The compute plane automatically creates this KAI resource on installation:

1
apiVersion: kai.scheduler/v1alpha1
2
kind: Topology
3
metadata:
4
  name: nvcf-mnnvl-topology
5
spec:
6
  levels:
7
    - nodeLabel: nvidia.com/gpu.clique
8
    - nodeLabel: kubernetes.io/hostname

Enabling addons.topologyAwareScheduling without addons.kaiScheduler.enabled stops Helmfile rendering with an error.

Example: place a StatefulSet in one GPU clique

Add the KAI topology annotations to the StatefulSet object:

1
apiVersion: apps/v1
2
kind: StatefulSet
3
metadata:
4
  annotations:
5
    kai.scheduler/topology: "nvcf-mnnvl-topology"
6
    kai.scheduler/topology-required-placement: "nvidia.com/gpu.clique"
7
spec:
8
  podManagementPolicy: Parallel
9
  replicas: 2

KAI waits until all replicas fit in one value of nvidia.com/gpu.clique. This is also a gang-scheduling request. See Gang Scheduling for atomic placement behavior and Grove or Dynamo workloads.

Creating a Topology does not change workloads that do not opt in.

Use Grove and/or Dynamo

Enable the complete scheduling stack during compute stack installation when functions use Grove or Dynamo:

1
addons:
2
  topologyAwareScheduling:
3
    enabled: true
4
  kaiScheduler:
5
    enabled: true
6
  groveOperator:
7
    enabled: true
8
  dynamoOperator:
9
    enabled: true

The topologyAwareScheduling toggle enables topology handling in Grove and automatically creates a binding for each configured KAI topology on installation:

1
apiVersion: grove.io/v1alpha1
2
kind: ClusterTopologyBinding
3
metadata:
4
  name: nvcf-mnnvl-topology-binding
5
spec:
6
  levels:
7
    - domain: gpuclique
8
      key: nvidia.com/gpu.clique
9
    - domain: hostname
10
      key: kubernetes.io/hostname
11
  schedulerTopologyBindings:
12
    - schedulerName: kai-scheduler
13
      topologyReference: nvcf-mnnvl-topology

See the Grove automatic MNNVL guide for more details.

Example: place a Grove workload in one GPU clique

Grove workload authors use the portable gpuclique domain from the generated binding instead of the cluster’s node label:

1
apiVersion: grove.io/v1alpha1
2
kind: PodCliqueSet
3
metadata:
4
  name: distributed-inference
5
spec:
6
  replicas: 1
7
  template:
8
    topologyConstraint:
9
      topologyName: nvcf-mnnvl-topology-binding
10
      pack:
11
        required: gpuclique
12
    cliques:
13
      - name: worker
14
        spec:
15
          roleName: worker
16
          replicas: 2
17
          minAvailable: 2
18
          podSpec:
19
            containers:
20
              - name: worker
21
                image: <worker-image>
22
                resources:
23
                  limits:
24
                    nvidia.com/gpu: "4"

Grove translates gpuclique to nvidia.com/gpu.clique and sends the constraint to KAI. Child PodClique or PodCliqueScalingGroup constraints can select the same or a narrower domain. See the Grove topology constraint rules.

Example: place an entire DynamoGraphDeployment in one GPU clique

This DynamoGraphDeployment will create Grove pod cliques for GPU and frontend workers that get scheduled in a single gpuclique domain:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: my-llm
5
spec:
6
  topologyConstraint:
7
    topologyProfile: nvcf-mnnvl-topology-binding
8
    packDomain: gpuclique
9
  services:
10
    VllmWorker:
11
      componentType: worker
12
      replicas: 2
13
      envFromSecret: hf-token-secret
14
      resources:
15
        limits:
16
          gpu: "1"
17
      extraPodSpec:
18
        mainContainer:
19
          image: my-image
20
          command: ["/bin/sh", "-c"]
21
          args:
22
            - python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B
23
    Frontend:
24
      componentType: frontend
25
      replicas: 1
26
      extraPodSpec:
27
        mainContainer:
28
          image: my-image
29
          command: ["/bin/sh", "-c"]
30
          args:
31
            - python3 -m dynamo.frontend

In general Dynamo creates Grove resources from a DynamoGraphDeployment, which Grove and KAI place. Before adding topology constraints to a DynamoGraphDeployment, see the topology-aware scheduling guide for Dynamo 1.2.1 to check the workload fields and topology resources expected by the compute plane stack’s pinned operator.

Verify topology resources (Cluster admins only)

Confirm that nodes have GPU clique labels:

$
kubectl get nodes -L nvidia.com/gpu.clique

Confirm that the KAI topology exists:

$
kubectl get topologies.kai.scheduler nvcf-mnnvl-topology

When Grove is enabled, confirm that its binding exists:

$
kubectl get clustertopologybindings.grove.io \
>
  nvcf-mnnvl-topology-binding

If Pods remain Pending, verify that one clique has enough free nodes for the entire gang. Then inspect the Pod and scheduler events:

$
kubectl describe pod <pod-name> -n <namespace>
$
kubectl describe podgroup <pod-group-name> -n <namespace>