Summary - RAG Deployment Best Practices#

For Enterprise RAG, this guide has covered how to deploy RAG components for scaling. The primary goal for Enterprise RAG is to enable AI inference applications for Chat, Summarization, Generation using Enterprise data (100K+ documents, 100M+ vectors) at Enterprise level scale (10K+ users)

Deploy RAG services (Milvus vector DB) and NIM microservices (LLM, Reranking, Embedding and Extractions) independently from the RAG blueprint and add the endpoints to the RAG blueprint Helm chart.

This provides flexibility to scale and manage the lifecycle independently, upgrade NIM and RAG blueprint (RAG server Langchain server) independently. It provides ability to use multiple LLM models with the RAG pipeline and dynamically update the RAG server with new endpoints if needed.

RAG Retrieval (Blueprint) Deployment#

The RAG blueprint is used to deploy the RAG retrieval pipeline, mainly the RAG server that is the data orchestrator for retrieval pipeline, configure the RAG retrieval parameters, along with NIM and Milvus vector DB endpoints. LLM NIM, Reranking, Embedding NIM microservices and Milvus vector DB are deployed independently.

LLM NIM Deployment#

  • Use NIM Cache: Deploy the LLM NIM independently from the RAG blueprint using NIM Cache + NIM Service (NIM Operator). NIM Cache stores the model on an NFS PVC (80 GB) and reduces scale-out time by 50-80% from ~25 min initial download to ~8-9 min for subsequent pod additions.

  • Single GPU Throughput Profile: Using Quantization NVFP4 on RTX PRO 6000 with vLLM engine, single GPU tensor parallelism TP=1, throughput profile; delivers 10%-25% better output throughput with equivalent accuracy vs. FP8.

GPU

Engine

Precision

TP

Profile

RTX PRO 6000

vLLM

NVFP4

TP1

throughput

H200 NVL

vLLM

FP8

TP1

throughput

  • Context Length Limit: Set NIM_MAX_MODEL_LEN: 16256 for RTX PRO 6000. For H200 NVL, this can be increased to 32768.

  • KV Cache Reuse: NIM_ENABLE_KV_CACHE_REUSE is enabled by default (NIM v1.14.0+). This caches the system prompt and significantly reduces TTFT for repeated queries.

  • NIM Cache GPU Profile: Specify gpus.product in NIM Cache to match the target GPU: RTX6000_BLACKWELL_SV or H200_NVL.

RAG Server Configuration#

  • Workers and Resources: Baseline: 32 workers, 8 vCPU, 32 GiB RAM. For High performance benchmarking scale to 64 workers, 16 vCPU, 64 GiB RAM. Insufficient workers directly limit the concurrency or maximum concurrent requests.

  • Service Exposure: Deploy the rag-server with a LoadBalancer service or Ingress service for external access.

  • Enable Metrics and tracing: Configure Prometheus ServiceMonitor for RAG server with the kube-prometheus-stack label. Enable OpenTelemetry tracing on the rag-server and identify service latency bottlenecks for RAG retrieval API calls => HTTP POST /v1/chat/completions

Performance Tuning - Disable Non-Essential Features#

Disable the following features for RAG Retrieval throughput benchmarking - to reduce the scope of the tests and components for scaling ratios and formula. Some components like Citations, Guardrails, Multiturn history, Thinking/Reasoning need to be added to production RAG retrieval pipelines and will add to the end-to-end latency.

Feature

Latency Impact

Env Var

Citations

Significantly increases output token size

ENABLE_CITATIONS: "False"

Query Rewriter

Adds a full additional LLM call (~doubles latency)

ENABLE_QUERYREWRITER: "False"

Guardrails

Adds additional NIM call overhead

ENABLE_GUARDRAILS: "False"

Multi-turn history

Increases prompt ISL by up to 5x

ENABLE_MULTITURN: "False"

Thinking/Reasoning

Generates 10-20x more output tokens

FILTER_THINK_TOKENS: "true"

Reflection Agent

Only required for accuracy benchmarking (adds LLM call)

ENABLE_REFLECTION: "False"

Observability#

  • Prometheus ServiceMonitors (Required for HPA): Enable Prometheus ServiceMonitors for all NIM microservices and Milvus. The release: kube-prometheus-stack label must match your Prometheus deployment. Without this, metrics will not be scraped and HPA scaling will not function.

  • OpenTelemetry Tracing: Enable OpenTelemetry tracing on the rag-server, Milvus, NV-Ingest, and all NIM microservices. Use Jaeger to visualize spans and identify per-component latency bottlenecks during load testing. OTEL collector endpoint: http://rag-opentelemetry-collector:4318.

  • Grafana Dashboards: Use Grafana dashboards to monitor GPU utilization, KV-cache usage, throughput (TPS), and latency (TTFT, ITL) across all services in real time.

Vector DB - Milvus Distributed Deployment#

The RAG Blueprint deploys Milvus in standalone mode (single GPU) by default – adequate for developer demos and datasets < 50K vectors. For enterprise deployments with millions of vectors and hundreds of concurrent users, deploy Milvus in Distributed (clustered) mode. Distributed mode allows Data/Index nodes and Query/Search nodes to be scaled independently based on ingestion and retrieval load.

Deployment Configuration

  • Helm Deployment: Add the Milvus Helm repo (zilliztech/milvus-helm) and deploy with a custom values file. Use the GPU-accelerated image: milvusdb/milvus:v2.6.5-gpu to enable cuVS (GPU CAGRA) for indexing.

  • Cluster Mode: Enable cluster mode (cluster.enabled: true). This activates the distributed architecture with separate coordinator, data, query, index, and proxy nodes.

Hybrid GPU-CPU Mode

  • GPU-Index + CPU-Search: Use GPU CAGRA (cuVS) for index building and convert to HNSW at collection load time for CPU-based search (enable adapt_for_cpu index parameter). This delivers 8-10x faster indexing than CPU with the cost efficiency of CPU search.

  • EF Parameter for CPU Search: Set the EF parameter (sent with each Milvus query) to

    = VDB TopK value. Example: if VDB TopK = 20, set ef = 20. Higher EF improves accuracy at the cost of slightly increased latency.

  • Indexing Performance Impact: GPU CAGRA indexing accounts for 70-80% of total ingestion time. GPU-accelerated indexing reduces indexing time by 6X-10X and end-to-end ingestion time by 2X-4X compared to CPU-only indexing.

Milvus data and query node resource allocation

Scale Data (index) and Query (search) nodes independently based on load:

Scale

Data Nodes

GPU (Data Node)

Data node [CPU, Memory]

Query Nodes

Query node [CPU, Memory]

Baseline 100K docs, 1M vectors

1

0.25 GPU mig-1g.24gb

[16 vCPU, 32GiB]

1

[16 vCPU, 40GiB]

Mid-scale 10M docs, 10M vectors

1

0.25 GPU per node

[16 vCPU, 32GiB] per node

2-4

[16 vCPU, 40 GiB] per node

Large Enterprise 10M docs, 100M vectors

2-4

0.5 GPU per node

[16 vCPU, 32GiB] per node

5-10

[16 vCPU, 40GiB] per node

  • Query Node RAM Sizing: Allocate ~10 GiB RAM per 1M vectors (2048 dim x FP32) to Query nodes. Scale Query nodes horizontally to handle simultaneous searches across multiple collections.

  • Data Node Scaling: Scale Data/Index nodes up during large ingestion runs (> 100K documents). Scale them back down after ingestion is complete to free GPU resources for retrieval.

  • Query Node HPA: Enable HPA for Query nodes (hpa.enabled: true) with 80% CPU and memory utilization targets. Set maxReplicas: 8 as a starting point for enterprise scale.

Storage Configuration

  • ETCD Storage (Critical): Milvus distributed requires 3X ETCD nodes with fast local NVME disks. For ETCD nodes, enable fast block storage (longhorn.io) with local NVME disks to avoid sync timeouts. Slow ETCD storage is the most common cause of Milvus instability and write timeouts.

  • MinIO Object Store: Allocate 500 GB+ NFS storage for MinIO (storageClass: nfs-client). For high-throughput ingestion, deploy MinIO in distributed mode (zones: 2, 2+ replicas).

  • Storage Classes: Use nfs-client for all Milvus components except ETCD. NFS provides the ReadWriteMany access mode required for shared PVCs in clustered deployments.