RAG Sizing Guidelines

In order to size the Enterprise RAG service, define the Peak Usage or maximum Concurrency or Concurrent Requests (CRs) that need to be supported for the use case (Chat, Summarization, Generation or Analysis/Research).

Concurrent Requests (CR): It indicates the maximum number of in-flight user requests for a given First Token Latency that can be supported by the Enterprise RAG service. It is a measure of Peak Usage at given Latency.

To size the workload first estimate what will be the peak number of active users that will use the RAG AI Chat service at any given time. And then size the RAG service to support that number of concurrent requests. Start conservatively and then refine the estimate; scale up with more users, add more servers and GPUs over time.

Next select the SLA response time or Latency constraint for the use case, e.g.

• Customer Service AI Chatbot: Low Latency, TTFT: <3s

• Meeting Summarization: Balanced Performance; TTFT: 10-20s

• Code Generation: Balanced Performance; TTFT: 10s-20s

• Research Analysis, Report Generation: Max. Throughput; TTFT: 30s-60s

For example; for a Large Enterprise with 5K engineers if IT creates a new AI chatbot for Engineering bug management; a good estimate of peak usage for the Chat service is 10% of total users; then size the RAG for AI Chat(128/128) service for 500 concurrent requests.

Next pick the SLA response time or Latency constraint; for an internal service tool for engineering, select Low Latency bucket with response within 3s (TTFT <3s).

To support RAG Chat(128/128), with Low Latency (TTFT <3s) and Peak Users CR=500, what is RAG or LLM Scale, number of nodes and GPU count that is needed. Assuming near linear scaling with RAG Scaling Efficiency E > 80%, this can be determined from the RAG Sizing for Chat table below.

Use the RAG Linear Scaling Table provided above to determine the number of GPUs and nodes needed to build the Sizing table below for each RAG Scale nX.

Peak Usage %	Concurrent Requests (CR)	Latency Threshold	RAG Scale	Reranker ON	GPUs and Nodes
Enterprise Size: Large, 5,000 Users. Peak 10% CR =500	500	Low Latency TTFT <3s	Scale 20X	ON	23.75 GPUs, 3 Node

Note

RAG Scale implies LLM Scale + scaling other components (Milvus Vector DB nodes, Reranker, Embedder) based on component Latency thresholds.

RAG Sizing for Chat 128/128

Assuming near linear scaling with RAG Scaling Efficiency E > 80%.

Peak Usage %	Concurrent Requests (CR)	Latency Threshold	RAG (LLM) Scale	GPUs
Enterprise Size: Small, 200 Users
Peak 10% CR =20	20	Balanced Perf. TTFT 3s-6s	Scale 1X	2.75 GPUs, 1 Node
Peak 10% CR =20	20	Low Latency TTFT <3s	Scale 1X	2.75 GPUs, 1 Node
Enterprise Size: Small, 500 Users
Peak 10% CR =50	50	Balanced Perf. TTFT 3s-5s	Scale 1X	2.75 GPUs, 1 Node
Peak 10% CR =50	50	Low Latency TTFT <3s	Scale 2X	3.75 GPUs, 1 Node
Enterprise Size: Medium, 1000 Users
Peak 10% CR =100	100	Balanced Perf. TTFT 3s-5s	Scale 2X	3.75 GPUs, 1 Node
Peak 10% CR =100	100	Low Latency TTFT <3s	Scale 5X	6.75 GPUs, 1 Node
Enterprise Size: Large, 5000 Users
Peak 10% CR = 500	500	Balanced Perf. TTFT 3s-5s	Scale 10X	12.75 GPUs, 2 Nodes
Peak 10% CR = 500	500	Low Latency TTFT <3s	Scale 25X	29.75 GPUs, 4 Nodes
Vary the Peak Usage Percentage
Peak 8% CR = 400	400	Balanced Perf. TTFT 3s-5s	Scale 8X	9.75 GPUs, 2 Node
Peak 6% CR = 300	300	Low Latency TTFT <3s	Scale 15X	17.75 GPUs, 3 Node
Peak 20% CR = 1000	1000	Max Throughput TTFT <10s	Scale 13X	15.75 GPUs, 3 Node
Enterprise Size: X-Large, 10,000 Users
Peak 20% CR = 2000	2000	Balanced Perf. TTFT 3s-5s	Scale 40X	46.75 GPUs, 6 Node
Peak 16% CR = 1600	1600	Low Latency TTFT <3s	Scale 80X	91.25 GPUs, 12 Node
Peak 20% CR = 2000	2000	Max Throughput TTFT <10s	Scale 25X	29.75 GPUs, 4 Node
Vary the Peak Usage Percentage
Peak 30% CR = 3000	3000	Balanced Perf. TTFT 3s-5s	Scale 60X	69.25 GPUs, 9 Node
Peak 18% CR = 1800	1800	Low Latency TTFT <3s	Scale 90X	103.25 GPUs, 14 Node

RAG Sizing for Summarization (1024/512)

Now let’s look at Summarization use cases.

For example: Summarizing meeting transcripts or financial reports or research papers. Latency is not as critical a requirement but the goal is supporting a large number of Peak users and providing response within reasonable time. For Summarization, pursuing Balanced Performance by optimizing Output Throughput (TPS) and Per User Throughput (TPS) is the most cost efficient solution.

Assuming near linear scaling with RAG Scaling Efficiency E > 80%.

Peak Usage %	Concurrent Requests (CR)	Latency Threshold	RAG (LLM) Scale	Reranker ON SU, Nodes, GPUs
Enterprise Size: Small, 200 Users
Peak 10% CR = 20	20	Balanced Perf. TTFT 9s-15s	Scale 1X	2.75 GPUs, 1 Node
Peak 10% CR = 20	20	Low Latency TTFT <10s	Scale 1X	2.75 GPUs, 1 Node
Enterprise Size: Small, 500 Users
Peak 10% CR = 50	50	Balanced Perf. TTFT 9s-15s	Scale 1X	2.75 GPUs, 1 Node
Peak 10% CR = 50	50	Low Latency TTFT <10s	Scale 2X	3.75 GPUs, 1 Node
Enterprise Size: Medium, 1,000 Users
Peak 10% CR = 100	100	Balanced Perf. TTFT 9s-15s	Scale 2X	3.75 GPUs, 1 Node
Peak 10% CR = 100	100	Low Latency TTFT <10s	Scale 4X	5.75 GPUs, 1 Node
Enterprise Size: Large, 5,000 Users
Peak 10% CR = 500	500	Balanced Perf. TTFT 10s-20s	Scale 10X	12.75 GPUs, 2 Node
Vary the Peak Usage Percentage
Peak 8% CR = 400	400	Balanced Perf. TTFT 10s-20s	Scale 8X	9.75 GPUs, 2 Node
Peak 5% CR = 250	250	Low Latency TTFT <10s	Scale 10X	12.75 GPUs, 2 Node
Enterprise Size: X-Large, 10,000 Users
Peak 10% CR = 1000	1000	Balanced Perf. TTFT 15s-30s	Scale 20X	23.75 GPUs, 3 Node
Vary the Peak Usage Percentage
Peak 20% CR = 2000	2000	Balanced Perf. TTFT 35s-60s	Scale 40X	46.75 GPUs, 6 Node