Gang Scheduling

Gang scheduling holds a group of Pods until the scheduler can place every required member. This prevents a partial deployment from consuming GPUs while the remaining Pods stay Pending.

NVCF Helm functions and tasks may use custom resources for several components for gang-scheduled workloads:

• KAI Scheduler provides queueing, resource allocation, and atomic placement through PodGroup resources.
• Grove represents related workload roles as PodCliqueSet, PodClique, PodCliqueScalingGroup, and PodGang resources. Grove delegates placement to KAI Scheduler.
• Dynamo describes inference services such as frontend, prefill, and decode workers in a DynamoGraphDeployment. The Dynamo operator uses Grove to orchestrate those services.

The NVCF Cluster Agent (NVCA) admits the KAI, Grove, and Dynamo resources in function Helm charts when their compute plane add-ons are enabled. NVCA also uses KAI implicitly when the KAIScheduler feature flag is enabled to binpack workloads.

For more background, see the upstream KAI gang scheduling guide and Grove core concepts.

Enable gang scheduling

Enable KAI Scheduler, Grove or Dynamo when functions use their custom resources:

1
addons:
2
  kaiScheduler:
3
    enabled: true
4
  groveOperator:
5
    enabled: true
6
  dynamoOperator:
7
    enabled: true

The compute plane installs the components in dependency order:

1. KAI Scheduler
2. Grove
3. Dynamo

The same add-ons configure NVCA:

• KAI adds the KAIScheduler feature gate and permits PodGroup.
• Grove permits PodCliqueSet, PodClique, PodCliqueScalingGroup, and PodGang. Grove requires KAI Scheduler to be enabled.
• Dynamo adds the DynamoOperatorSupport feature gate and permits Dynamo custom resources. Dynamo requires Grove to be enabled.

See KAI Scheduler for queue configuration and the standalone installation path.

Gang schedule a StatefulSet with KAI

Use a StatefulSet with parallel Pod management for a direct KAI workload:

1
apiVersion: apps/v1
2
kind: StatefulSet
3
metadata:
4
  name: distributed-worker
5
spec:
6
  podManagementPolicy: Parallel
7
  replicas: 2
8
  selector:
9
    matchLabels:
10
      app: distributed-worker
11
  template:
12
    metadata:
13
      labels:
14
        app: distributed-worker
15
    spec:
16
      containers:
17
        - name: worker
18
          image: <worker-image>

When the KAIScheduler feature is enabled, NVCA assigns the KAI scheduler and queue to the Pod template. KAI creates one PodGroup for the StatefulSet and waits until every replica can be placed.

Add topology annotations only when the gang must also fit in a specific hardware domain. See Topology-Aware Scheduling.

The multi-node Helm function sample contains optional KAI topology annotations for gangs that also need GPU clique placement.

Gang schedule a Dynamo workload with Grove

A DynamoGraphDeployment groups inference roles in one resource. A multinode service tells Dynamo and Grove that each replica needs multiple Pods:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: myllm
5
spec:
6
  services:
7
    VllmDecodeWorker:
8
      componentType: worker
9
      replicas: 1
10
      multinode:
11
        nodeCount: 2
12
      resources:
13
        limits:
14
          gpu: "4"

The Dynamo operator converts the service into Grove resources. Grove groups the Pods, and KAI reserves and places the complete group. Function authors do not explicitly create PodGang or PodGroup resources for a DynamoGraphDeployment.

See the Dynamo Operator sample for a disaggregated frontend, prefill, and decode workload. See the upstream Dynamo multinode guide and Grove quickstart for advanced workload configuration.

Verify gang scheduling (Cluster admins only)

Check that the operators are running:

$
kubectl get pods -n kai-scheduler
$
kubectl get pods -n grove-system
$
kubectl get pods -n dynamo-system

Inspect the scheduling resources for a deployed function:

$
kubectl get podgroups.scheduling.run.ai -A
$
kubectl get podgangs.scheduler.grove.io -A
$
kubectl get podcliquesets.grove.io -A

If a gang remains Pending, describe its Pods and KAI PodGroup. Common causes are insufficient free GPUs, queue limits, or topology constraints that no available domain can satisfy.

$
kubectl describe pod <pod-name> -n <namespace>
$
kubectl describe podgroup <pod-group-name> -n <namespace>