KAI Scheduler Integration Guide
KAI Scheduler is an open source Kubernetes scheduler for AI workloads. NVCF integrates with KAI to place GPU workload Pods. With the configuration in this guide, KAI bin-packs those Pods onto eligible nodes. This keeps whole nodes free for larger workloads and improves cluster utilization. Helm functions can also request gang scheduling and topology-aware placement in their charts. See Gang Scheduling and Topology-Aware Scheduling.
KAI coexists with the default Kubernetes scheduler. When the KAIScheduler
feature gate is enabled, the NVIDIA Cluster Agent (NVCA) assigns NVCF workload
Pods to KAI. Other Pods can continue using the default scheduler. When the
feature gate is disabled, all Pods use the default scheduler, even if KAI is
installed.
Operational responsibilities
The platform operator owns the KAI installation and lifecycle. NVCA manages its integration with KAI, not the KAI service itself.
The compute plane stack can automate KAI installation and configuration. This does not transfer lifecycle ownership to NVCA.
Install KAI Scheduler
Use a tested KAI Scheduler release that is compatible with your NVCF compute plane stack.
Use the compute plane stack
Set the following in your nvcf-compute-plane Helmfile environment:
The add-on installs KAI as release and namespace kai-scheduler, configures
its default queues, and enables NVCA’s KAIScheduler feature gate unless
explicitly disabled with -KAIScheduler. Apply the environment through your
compute plane installation workflow. Skip the manual installation below.
The add-on manages the Helm release kai-scheduler in the kai-scheduler
namespace. If a release with that name exists, enabling the add-on upgrades
it to the version pinned by the compute plane stack and applies the stack’s
values. A KAI installation under another release name or namespace is not
adopted. Remove it before enabling the add-on, or keep the add-on disabled
and follow the next section.
Use an existing or separately managed installation
If KAI is managed outside the compute plane stack, leave the installation add-on disabled and configure KAI with the values below. Do not install a second KAI release.
NVCA expects a parent queue named default-parent-queue and a child queue
named default-queue. Other queues may also exist.
Set unlimited (-1) quotas and limits on the queues used for NVCF workloads.
NVCA relies on this configuration for cluster capacity and usage tracking.
If NVCF and non-NVCF workloads share a cluster with limited KAI queues,
enable Shared Cluster mode so NVCA
excludes non-NVCF nodes from capacity tracking and scheduling. Nodes running
NVCF workloads must be labeled nvca.nvcf.nvidia.io/schedule=true.
Create values.yaml with the required scheduler and queue settings:
values.yaml
From KAI v0.18, the chart turns GPU sharing on unless global.gpuSharingMode
is set. NVCF workloads request whole GPUs, so set it to Disabled.
The preempt, consolidation, and reclaim actions are disabled so that KAI never
evicts running NVCF workloads to place other workloads. Reclaim needs the most
care. Because NVCF queues have unlimited quotas, quota-based reclaim never takes
from them. With KAI’s default allow-consolidating-reclaim, however, reclaim
can still move running NVCF Pods to other nodes to make room for another
queue’s pending gang. Workloads that create their own queues, such as Dynamo,
make this possible.
KAI’s stalegangeviction action stays enabled. It evicts the remaining Pods of
a gang that stays below its minimum size for longer than the staleness grace
period (60 seconds by default). Grove’s kai-scheduler
backend, which Grove enables by default from v0.1.0-alpha.13, owns this
decision for Grove workloads, including Dynamo’s. Where that backend is active,
add default-staleness-grace-period: "-1s" under scheduler.args, as the
Dynamo 1.5 upgrade notes require. Keep the default when KAI runs without Grove
or when Grove’s kai-scheduler backend is disabled. When the compute plane
stack installs KAI, it sets this whenever the Grove operator add-on is enabled.
The setting changes the scheduler-wide default, so it applies to every PodGroup
that does not set its own grace period, including PodGroups owned by other
teams on a shared cluster. Such a gang keeps its running Pods for as long as it
stays below its minimum size. A workload that needs stale-gang eviction can set
its own grace period with the kai.scheduler/staleness-grace-period annotation
on its Pods or its parent resource. KAI’s podgrouper copies the annotation into
the PodGroup it creates. For a PodGroup that the workload creates itself with
kai.scheduler/skip-podgrouper, set spec.stalenessGracePeriod on the
PodGroup instead.
If other teams on a shared cluster rely on reclaim to enforce their queue
quotas, use the following scheduler block instead of the one in the values
above. It keeps reclaim enabled but turns off allow-consolidating-reclaim, so
reclaim no longer moves running NVCF Pods, and quota-based reclaim between the
other queues still works. This relies on the proportion plugin’s
queuePriorityInQuotaReclaim argument staying at its default of false. When
it is enabled, a queue with a higher priority can reclaim in-quota resources
from a lower-priority queue, which includes evicting preemptible NVCF Pods.
Where Grove’s kai-scheduler backend is active, keep both arguments by
uncommenting the staleness setting:
For a new installation, replace <kai-version> with the version of the
kai-scheduler release in
deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl.
Read this file from your selected compute plane release tag, not from main.
For an existing KAI release, apply the same values with helm upgrade:
After KAI is ready, add KAIScheduler to the existing NVCA feature-gate list.
For standalone NVCA Helm values, use selfManaged.featureGateValues.
For compute plane environment values, use
global.nvcaOperator.selfManaged.featureGateValues.
Preserve the other feature gates when updating the list. See
Managing Feature Flags.
Installing KAI alone does not enable NVCF to use it. Both the KAI installation and the NVCA feature gate are required.
Verify the integration
Check the KAI components and default queues in the compute cluster:
After deploying a function, inspect its workload Pods:
New NVCF workload Pods should use kai-scheduler and default-queue.
If a Pod remains Pending, see Troubleshoot the integration.
Troubleshoot the integration
When the KAIScheduler feature gate is enabled, NVCA checks the KAI queue
configuration and reports the result as the kai-scheduler-queues health
component. A failed check marks the agent unhealthy.
Check the agent status:
If HEALTH is not healthy, read the component detail from the NVCA
health endpoint:
If the component is healthy but workload Pods stay Pending, the queue
configuration is correct. Confirm that the Pod uses kai-scheduler and that
the KAI components are running, then check the Unschedulable event on the
Pod and the KAI scheduler logs:
Maintain KAI Scheduler
- Monitor KAI component readiness and pending workload Pods.
- Validate KAI upgrades with your NVCA and compute plane versions before applying them to production. Follow the selected KAI release’s upgrade guidance.
- After an upgrade, verify the queues and deploy a test function to confirm scheduling, readiness, and invocation.
Pods assigned to KAI do not automatically fall back to the default scheduler. If KAI is unavailable, new or replacement workload Pods can remain pending. The platform operator is responsible for restoring KAI availability.
Schedule multi-Pod workloads
KAI can hold a multi-Pod workload until all required members fit. Grove and Dynamo build on this behavior for multi-role inference services.
A Helm function requests this in its chart. NVCA passes the chart’s KAI,
Grove, and Dynamo resources through when the matching add-ons are enabled.
It does not create PodGroup resources or set minMember on a workload’s
behalf. See Gang Scheduling for add-on setup,
workload examples, and troubleshooting.
On NVLink-optimized clusters, KAI can also place the complete gang in one GPU clique. See Topology-Aware Scheduling for GPU DRA prerequisites, topology configuration, Grove bindings, and function examples.