Introduction#
To deploy Enterprise-grade RAG in a production environment, enterprises need a certified hardware reference architecture and a validated software stack where the RAG workload can be reliably deployed at scale for optimal performance. The RAG Blueprint integrates multiple NVIDIA NIM microservices as well as open source and third party software microservices. These RAG components need to be deployed separately and fine tuned to scale in large clusters of up to 32 nodes. This Enterprise RAG guide provides deployment recommendations for the RAG services in the ingestion and retrieval pipelines at Baseline scale and at high performance scale.
Hence there is a need for end-to-end validation of the RAG blueprint on the NVIDIA AI Factory RTX PRO with NVIDIA RA 2-8-5 hardware cluster with Enterprise RA Kubernetes platform stack (including BCM, NVIDIA AI Enterprise operators, Kubernetes, storage, networking, observability and metrics) for both functionality and optimal performance.
Enterprises will need to determine how to right size and scale the RAG deployment to meet the growing needs of their users. They need to know what parameters to tune, KPI metrics to track and which component (LLM, Reranker, Vector DB) component to scale up or scale out when they hit a bottleneck. They need to size their RAG solution; determine how many servers and GPUs are needed based on the:
Usecase: Chat or Summarization or Generation; this is defined by context length; ISL and OSL combination
Number of Concurrent Requests (CR) and
Maximum latency: Latency can vary for each usecase
Scope#
This guide provides best practices on how to deploy, scale and size RAG for Enterprise deployments on Enterprise RA 2-8-5-200 with GPUs: RTX PRO 6000 and H200 NVL GPUs on a vanilla Kubernetes cluster.
Deploying RAG for Enterprise based on NVIDIA RAG Blueprint; identify potential bottlenecks; provide sufficient resources to scale to thousands of concurrent requests.
Ingesting Data into a Vector database (VDB): How to deploy, scale and fine tune NeMo Retriever Extraction (GitHub) and the Vector DB (Milvus) index; How to deploy Milvus vector DB with NVIDIA cuVS for GPU accelerated indexing (CAGRA) and CPU for search; understand the implication of chunk size and Vector DB size; How to bulk ingest thousands of documents; How to measure ingestion performance with large amounts of data.
Retrieving data with multiple Concurrent Requests (CR) - What parameters to tune for the RAG server (Langchain server), Large Language Models (LLM) and other NeMo Retriever NIM microservices (embedding, reranking); How to define the number of chunks (TopK) to retrieve from the Vector database (VDB); Whether to enable the Reranking NIM; When to scale Reranker and Milvus search (query) nodes; How to benchmark retrieval performance with key metrics for latency and throughput.
Scaling and Sizing workloads on Enterprise RA: What are the benchmarking tools, methodologies and KPI (key performance indicators) metrics for scaling one or more RAG components (LLM, NIM microservices, VDB) on the Enterprise RA 2-8-5-200 cluster; while scaling from 4 to 16 servers and 32 to 128 GPUs; How to size workloads based on number of Concurrent Requests (CR) along with latency constraints and throughput requirements.
The RAG deployment provides best practices for deploying Enterprise-grade RAG ingestion and retrieval pipelines in production environments.
NVIDIA 2-8-5-200 GPU Node and NVIDIA SpectrumX Platforms Enterprise Reference Architecture (ERA)
Enterprise RAG Overview#
For Enterprise RAG, this guide leverages NVIDIA RAG Blueprint which provides an end-to-end RAG solution that is fully customizable and easy to deploy for AI developers using docker compose and for enterprise AI inference in Kubernetes clusters. It is available from NVIDIA NGC Catalog as a Helm chart that deploys multiple microservices (RAG server, Nemotron Text NIM microservices, Nemotron Extraction: NIM microservices and NV-Ingest, Milvus Vector Database, AI Agents, LLM, Observability tools for monitoring and tracing; Prometheus, Open Telemetry and Zipkin) in a Kubernetes cluster.
See table below for RAG workload baseline requirements:
RAG service |
RAG service summary |
Minimum Compute GPU/CPU/RAM |
|---|---|---|
Enterprise RAG |
Summary of all RAG ingestion and retrieval services: rag-server, NIM LLM, Embedding, Reranking, NeMo Retriever Extraction NIM microservices, NV-Ingest, Milvus Vector DB |
Baseline Total: 4 X GPUs, 40 vCPU, 200 GiB |
RAG Blueprint |
RAG Blueprint (github v2.5.0 / v2.4.0), 1 rag-server |
8 vCPU, 32GiB |
NIM LLM |
NIM LLM (Large Language Model), 1 NIM LLM llama-3.3-nemotron-super-49b-v1.5 |
RAG Retrieval only: 1 GPU, 1 vCPU, 64 GiB |
Nemotron Embedding NIM |
Text Embedding NIM, 1 Embedding NIM (2048 dims X FP32), llama-nemotron-embed-1b-v2 |
Run:ai / MIG: 0.5 GPU, 2 vCPU, 8 GiB |
Nemotron Reranking NIM |
Text Reranking NIM, 1 Reranking NIM, llama-nemotron-rerank-1b-v2 |
RAG Retrieval only: 1 GPU, 2 vCPU, 8 GiB |
NeMo Retriever Extraction (NV-Ingest) NIM microservices |
NeMo Retriever Extraction, v26.3.0 / v26.1.2 (github), 1 NV-Ingest microservice, 1 Redis DB and 4 extraction NIM microservices for multimodal data extraction: nemotron-ocr-v1, nemotron-page-elements-v3, nemotron-table-structure-v1, nemotron-graphic-elements-v1 |
RAG ingestion only: Run:ai: 1 GPU, MIG: 1.25 GPU, 16 vCPU, 44 GiB |
Vector DB Milvus or Elastic |
Milvus - GPU for index and CPU for search, Data: 1M embeddings (2048 dims X FP32), 1 Data/Index node (0.25 GPU, 4 vCPU, 8 GiB), 1 Query node (4 vCPU, 10 GiB), Object store: Minio (500 GB disk storage), ETCD (fast block storage 50GB), Pulsarv3 (message bus) |
Run:ai / MIG: 0.25 GPU, 16 vCPU, 32 GiB |
Table 1. Scope of Workloads and Resources
Intended Audience#
This guide is to support NVIDIA partners architecting Enterprise RAG solutions on Kubernetes production clusters and provide sizing recommendations and scaling best practices for 16-32 node cluster level deployments. The guide can be used for both, new and existing Enterprise RAG deployments to determine capacity and scale needed based on the usecase. This guide is targeted to the following Personas as they architect their Enterprise RAG solutions.
Enterprise Architects: Enterprise Architects tasked in designing and defining servers, GPU’s and Networking gear to determine what Infrastructure resources will be needed to support specific workloads around LLM’s based on NIM microservices.
MLOps Engineer: MLOps Engineers can use this to define Infrastructure requirements as they talk to Infrastructure/Cloud teams to carve out resources to run Inference Services on LLMs based on NIM microservices.
Platform Engineer: Platform Engineer can use this guide to determine how to design their Container Environment around Kubernetes and also what kind of resources will be needed by the cluster to support various LLM workloads based on NIM microservices.