Summary

NVIDIA Run:ai can be used to dynamically allocate GPUs to AI workloads in a Kubernetes Cluster. Run:ai can automatically scale up or scale down workload pods in a cluster, thereby releasing the amount of GPUs consumed. These free GPUs can then be assigned to other users/projects/workloads to drive the overall utilization of existing GPUs and datacenter resources. Run:ai can help scale NIMs based on concurrent users, user latency, and throughput. Below are the observations and conclusions for running NIM workloads at scale on a 16-node cluster with Run:ai.

• The Run:ai scheduler does not add any additional performance overhead to Inference workloads when running on full GPUs. The performance data for NIMs running with and without Run:ai are comparative.

• NIM scheduled on Fractional (0.5) GPUs

  • Scaled Linearly

  • Performance of CCU, TTFT, and Throughput was comparable to non-Run:ai tasks til 32X Scale. This is because if there is no other competing workload on that GPU, Run:ai will dynamically expand the GPU to full for the workload.

  • At peak scale, consuming all the 64 GPUs in a cluster, the no. of concurrent users and throughput recorded were slightly lower, the Throughput dropped by ~20%, and the Concurrent Users dropped by ~10%

• Enterprise IT can use Run:ai to load more than one NIM for the same number of GPUs and get a variety of models to run, as well as twice as many users, decreasing the overall TCO. However, at full scale 64X, the TTFT can drop by 3X and the Throughput by 0.4X.