Benchmarking and Scale Methodology#
NVIDIA AI-Q is an agentic solution based on the NVIDIA NeMo™ Agent Toolkit. NVIDIA NeMo Agent Toolkit is an open-source framework for building, profiling, and optimizing agents and tools for agentic workflows. NeMo Agent Toolkit also has a profiling and benchmarking tool that can be used to determine how an agentic workflow is performing on a given architecture. For this guide, we used the NeMo Agent Toolkit Sizing and profiling function to test simultaneous concurrent users requesting deep research reports on various kinds of topics. The profiler in the NeMo Agent Toolkit, collects usage statistics in real time. These stats include the time each session/user took to get the detailed report back from AI-Q, what the LLM Latency was compared to the overall Workflow latency. NeMo Agent Toolkit provides the following metrics that tells how a agentic workflow is performing at a given concurrency level (concurrent users utilizing the workflow)
The P95 LLM Latency (95th percentile LLM latency) column contains the latency, in seconds, across all LLM invocations. If multiple models are used, the value will trend towards the latency of the model with the highest latency.
The P95 WF Runtime (95th percentile workflow runtime) column contains the response time, in seconds, of the workflow and is computed across all runs at the specified concurrency.
The Total Runtime column contains the total time, in seconds, taken to process the entire dataset at a specified concurrency level.
NeMo Agent Toolkit sizing tool can be given how many concurrent users to run, here is a sample out put
Note
AI-Q also has man in the middle function, this function helps a user refine the created report further to their liking, the benchmarking tests done here did not test this feature.
Setup NeMo Agent Toolkit to benchmark AI-Q#
On a linux system that’s outside the Enterprise RA cluster, setup NeMo Agent Toolkit Clone the NeMo Agent Toolkit repository to the benchmarking system
git clone -b main https://github.com/NVIDIA/NeMo-Agent-Toolkit.git nemo-agent-toolkit
cd nemo-agent-toolkit
Initialize, fetch, and update submodules in the Git repository
git submodule update --init --recursive
Create a Python environment
uv venv --python 3.12 --seed .venv
source .venv/bin/activate
Install the NeMo Agent Toolkit library. To install the NeMo Agent Toolkit library along with all of the optional dependencies. Including developer tools (–all-groups) and all of the dependencies needed for profiling and plugins (–all-extras) in the source repository, run the following:
uv sync --all-groups --all-extras
In addition to plugins, there are optional dependencies needed for profiling. To install these dependencies, run the following:
uv pip install -e '.[profiling]'
Load Python modules needed for NeMo Agent Toolkit to talk to AI-Q Workflows
uv pip install --no-deps -e ../aiq-research-assistant/aira
Getting Started With Sizing a GPU Cluster#
To begin, set the configuration file and output directory.
cd <NAT-root-directory>
export CALC_OUTPUT_DIR=.tmp/sizing_calc/
export CONFIG_FILE=${CALC_OUTPUT_DIR}aiq-test.yml
mkdir -p ${CALC_OUTPUT_DIR}
Datasets needed for prompt during sizing#
The sizing tool config file, needs a dataset that has a list of prompts that it can use to run concurrent users tests. This is a JSON format file and can have any no. of entries. For benchmarking we used the dataset.json file in Appendix C. This file needs to be copied into the NeMo Agent Toolkit folder under .tmp directory
Configuration File for NeMo Agent Toolkit to run sizing#
We need to add a evaluation config file to let NeMo Agent Toolkit know where the workflow is, what are the different LLMs to test, how to access the LLM, and what datasets to use, please add a YAML config file called aiq-test-rtxpro6k.yml in the .tmp/sizing_calc folder in NeMo Agent Toolkit. This file, provided in Appendix D has the evaluation sections, please update the config file with the right IP/endpoints for NIM-LLM from RAG, the Instruct-LLM from AI-Q and the RAG server URL. Also, if needed update the Phoenix app endpoint to send traces for the benchmarking tests
Run Benchmarking#
Activate virtual environment
cd ~/tmp/NeMo-Agent-Toolkit
source .venv/bin/activate
uv pip install --no-deps -e ../../aiq-research-assistant/
Export Tavily API key
export TAVILY_API_KEY="tvly-prod-redacted"
Run The Sizing calculator
The example below shows a concurrency run for 1 and 5, this can be changed or more concurrencies added to the command
nat sizing calc --config_file .tmp/sizing_calc/aiq-test-rtxpro6k.yml --calc_output_dir .tmp/sizing_calc/concurrency_era_1_5 --concurrencies 1,5
Scale Methodology#
To establish the benchmarking on RTX PRO 6000 BSE GPUs, we started load testing AI-Q with a 1X scale of all the components, this means on the 9 node cluster, every component deployed between RAG and AI-Q was running at 1X scale. We started NeMo Agent Toolkit sizing to get NIM Latency and Workflow Latency at different incremental concurrency ranges at 1X scale, the concurrency we ran was from 2, 4, 8 etc till 256 users. We would then scale the reasoning NIM to 2X scale, doubling the total no. of pods and hence GPU consumption and fetch the NIM Latency and Workflow latency at the same concurrencies. The goal was to make sure the LLM Latency and the Workflow latency dropped as we scaled the reasoning NIM LLM pods. We did this till we maxed out all the GPUs in the cluster at 32X NIM LLM scale, where all the 72 GPUs we consumed.
Table 3: Scale and Benchmarking runs tested
Overall Scale |
Reasoning NIM Scale (Nemotron Super 49B) |
Instruct NIM Scale (Meta Lama 70B) |
Precision |
Other RAG Components Scale |
Concurrency |
|---|---|---|---|---|---|
1X |
1 Pod (2 GPUs) |
1 Pod (1 GPU) |
fp8 |
1X |
2,4,8,16,32,64,96,128,256 |
2X |
2 Pod (4 GPUs) |
1 Pod (1 GPU) |
fp8 |
1X |
2,4,8,16,32,64,96,128,256 |
4X |
4 Pod (8 GPUs) |
1 Pod (1 GPU) |
fp8 |
1X |
2,4,8,16,32,64,96,128,256 |
8X |
8 Pod (16 GPUs) |
1 Pod (1 GPU) |
fp8 |
1X |
2,4,8,16,32,64,96,128,256 |
16X |
16 Pod (32 GPUs) |
1 Pod (1 GPU) |
fp8 |
1X |
2,4,8,16,32,64,96,128,256 |
32X |
32 Pod (64 GPUs) |
1 Pod (1 GPU) |
fp8 |
1X |
2,4,8,16,32,64,96,128,256 |
Other Scale Options Tested |
|||||
1X |
1 Pod (2 GPUs) |
2 Pod (2 GPU) |
fp8 |
1X |
2,4,8,16,32,64 |
1X |
1 Pod (2 GPUs) |
1 Pod (1 GPU) |
fp8 |
Reranker 2X |
2,4,8,16,32,64 |
1X-32X |
1 Pod (2 GPUs)-32 Pod (54 GPUs) |
1 Pod (1 GPU) |
nvfp4 |
1X |
2,4,8,16,32,64 |
We also ran scaling exercises where we would scale the Instruct LLM instead on the Reasoning NIM to check the overall impact at different concurrencies. We even scaled Nemotron reranker in RAG’s pipeline to test how that impacts the overall latency of the workflow. Earlier benchmarking tests were done with bf16 precision as well.
Scale the reasoning NIM LLM from 1 X to 2X, please wait till all the pods are online.
kubectl scale statefulset rag-nim-llm --replicas=2 -n rag
Check if all the NIM-LLM pods are running
Kubectl get pods -n rag