Benchmarking and Scale testing Results for AI-Q

Performance tests were conducted using different numbers of concurrent users. The results, shown below, were generated by scaling various components from both the RAG and AI-Q blueprints. A few observations based on the runs

• On an average, each concurrent user call in AI-Q was generating ~17,000 -24,000 output tokens per session, so the overall ISL/OSL for AI-Q was 20/20000, see Figure 6 from the Phoenix server capturing spans and tokens

• AI-Q does at least two rounds of reasoning rounds to check for relevancy of the output tokens generated, see Figure 7 capturing the overall workflow

• We initially used precision of bf16 and moved to fp8 precision for NIM this improved the overall latency by ~20%

• When a large number of requests come to the Nemotron Super 49B NIM pod, the backend service of the pod limits the total number of requests to 4 for latency profile, and to 5 for throughput profile, this is cause of the optimizations done on the NIM for RTX PRO 6000 GPUs, keeping the host framebuffer memory in mind and also the KV cache usage needed to store context. As a result of this, few requests are waiting in the queue to process. The results here capture the p95 latency numbers, however the average latency is almost half the p95 latency. For example, the requests that get in the first batch complete the workflow in 6 minutes, however the requests that are in queue can take ~17-20 minutes.

Figure 5 Phoenix app showcasing the average tokens per session and P50/P95 latencies

Figure 6 Gantt chart showcasing where the workflow spends time

Latency Impact of Reasoning Model Scale

The Nemotron Super 49B reasoning model significantly influences both overall workflow latency and NIM LLM latency. Scaling the Reasoning Model NIM resulted in a latency reduction of approximately 10-30% for the same number of concurrent users. This is when the precision of the model is already set to fp8. Most other components part of the RAG and AI-Q blueprint did not reach max usage when running benchmarking tests. See Figure 8 for the overall GPU consumption of various components in the system. While AI-Q Nemotron Instruct NIM, was also high in usage, it did not contribute primarily to the overall workflow latency.

Figure 7 Overall GPU usage of various components

Latency v/s Concurrent Users at Scale

As we scaled the reasoning LLM pods from 1X till 32X, the overall workflow latency kept increasing as more concurrent users were added. Concurrent users at 96 were above the knee for latency. Concurrent users above 96 showed a higher drop in latency. See Figure 9. for reference. Overall, at 32X the workflow latency was lower by approximately ~30 % than at 2X scale.

Figure 8 Workflow Latency v/s Concurrent Users at scale

AI-Q scales Linearly

The AI-Q system demonstrated linear scalability with the Reasoning LLM NIM. Specifically, when the Reasoning LLM NIM was doubled (2X scale), the system could simultaneously handle twice the number of concurrent users—up to eight—while maintaining the workflow latency below 1000 seconds for individual users generating reports. This established that AI-Q’s scaling performance is directly proportional to the system’s scaling. Refer to Figure 10 for the benchmarked concurrent user capacity achieved under the 1000-second workflow latency constraint.

Figure 9 Linear concurrent users with System Scale

Latency Drops as Systems Scale

The performance analysis demonstrated a clear correlation between the scaling of the Reasoning Large Language Model (LLM) and a reduction in critical latency metrics. Specifically, for any defined level of concurrent users, increasing the Reasoning LLM resulted in a measurable drop in both the overall LLM processing latency and the end-to-end Workflow latency.

Figure 10 Workflow Latency dropped for a specific concurrency at scale