Inference Performance and Scale Results with Run:ai

Benchmarking without Run:ai

To determine the baseline performance of the two models we ran a single GPU benchmarking test with Gen-AI Perf, we create a NIM LLM service without Run:ai with a single pod and single GPU, incrementally increased the concurrent users loaded by Gen-AI Perf registering the TTFT and Throughput until we saw the TTFT coming close to 1000ms. We registered the concurrent users count at which this was achieved and then scaled the GPUs consumed by the NIM LLM Service by scaling the Pods incrementally by 16X, 32X, and 64X. We noted the Baseline and scale numbers. Based on the results, the no. of Concurrent users scaled linearly as we scaled the GPUs consumed by the NIM Workloads.

Below are the Baseline performance numbers recorded for each model for Single GPU.

Table 5: Single GPU Performance Numbers without Run:ai

NIM LLM	GPU/Pod	Pods	CCU	Throughput (Output Tokens/Sec)	TTFT (ms)
Meta Llama 3.1 8B	1	1	141	3059	971
Deepseek R1 Distill Llama 8B	1	1	135	3117	976

Benchmarking with Run:ai using a full GPU

Once we had baseline performance numbers without Run:ai scheduling the workloads on the cluster, we created a NIM Inference workload using Run:ai, ran the same test on a single GPU managed by Run:ai and then scaled the workloads to 16X, 32X and 64X. This would max out all the GPUs in the cluster. Gen-AI Perf tool was used to load as many concurrent users on the NIM service to reach a TTFT of close to 1000ms.

The results in terms of the total concurrent users and the Throughput matched or was close to that of the tests without Run:ai schedular. See below table:

Table 6: Single GPU Performance Numbers with Run:ai

NIM LLM	GPU/Pod	Pods	CCU	Throughput (Output Tokens/Sec)	TTFT (ms)
Meta Llama 3.1 8B	1	1	137	2977	987
Deepseek R1 Distill Llama 8B	1	1	141	3225	987

Benchmarking with Run:ai using a Fractional GPU

Run:ai supports allocating fractional GPUs to AI workloads, however when an AI workload runs and is requesting a fractional GPU, if there are no other workload competing for the fractional GPU, Run:ai can allocate the entire GPU to the workload dynamically, for the next set of tests we ran the same NIM using Run:ai, however, we allocated only half (0.5) the GPU of H100NVL to the NIM pod. We then ran the same Gen-AI perf tests on the NIM service and scaled the pods to 16x, 32X and 64X. Each Pod consumed 0.5 GPU and at 64X scale there was 1:1 mapping to pods and GPUs in the cluster.

Since there were no other competing workloads, the dynamic GPU allocation in Run:ai gave the pods the entire GPU, and hence the perf results for the fraction (0.5) GPU are comparable to the NIM running with a full GPU.

Table 7: Single GPU Performance Numbers with Run:ai fractional (0.5) GPU

NIM LLM	GPU/Pod	Pods	CCU	Throughput (Output Tokens/Sec)	TTFT (ms)
Meta Llama 3.1 8B	0.5	1	137	3086	995
Deepseek R1 Distill Llama 8B	0.5	1	141	3237	978

Performance of NIM LLM at Scale with Run:ai

The Below charts show how the NIM LLM scales with and without Run:ai (both fractional and full GPUs). The concurrent users and throughput scales linearly. At Max scale (64 GPUs) there was a slight decrease in performance of the NIM LLM for the 0.5 fractional GPU, it was around ~10% drop in total concurrent users supported and ~20% drop in throughput.

Below are the Perf numbers for Meta Llama 3.1 8B. We ran the same test on Deepseek R1 Distill Llama 8B; the results were similar.

Figure 17 Concurrent Users scale for Meta Lama 3.1 8B

Figure 18 Throughput scale for Meta Lama 3.1 8B

Simultaneous Multiple NIMs with Run:ai

So far, we have run individual NIMs with different models on the cluster. Since Run:ai supports running AI workloads on fractional GPUs, we load both the NIMs, Meta Llama 3.1 8B and Deepseek R1 Distill Lama 8B simultaneously using Run:ai and gave 0.5 /half GPU to each. The cluster was then targeted by Gen-AI Perf to run workloads with the same number. In the CCU, we registered for individual runs of the respective models. The individual NIM workload was incrementally scaled along with the NIM Service pods to 16X, 32X, and 64X to completely utilize all the 64 GPUs in the cluster. Each Pod consumed 0.5 GPU and at 64X scale there was 2:1 mapping of pods and GPUs in the cluster.

Both the Inference services scaled linearly with Run:ai, both the Concurrent users and Throughput scaled Linearly.

Figure 19 Concurrent User on Meta Llama 3.1 8B across the system

With Run:ai’s scheduler we were able to load twice as many users on the overall system, during peak workload where 128 pods (64 pods for each NIM service) were requesting 0.5 GPU each, maxing out the 64 GPUs, there was 3X drop in TTFT and around 0.8X drop in the total Throughput the cluster cloud handle.

The overall users we could load on the cluster were 2X more than what we could load with individual models when all the GPUs were consumed.

The System Throughput, or the total tokens processed by the cluster at max capacity, dropped by 0.4X compared to the total system throughput when individual NIMs were run at max GPU utilization.

Figure 20 Throughput on Meta Llama 3.1 8B across the system

The overall User Latency, or TTFT, increased by 3X at the max GPU utilization; however, comparatively, the total no. of users that could be loaded and could run Inference on multiple models doubled.

Figure 21 TTFT on Meta Llama 3.1 8B across the system