Enterprise RAG Retrieval Scaling and Sizing Guide

NVIDIA Enterprise Reference Architecture

• Abstract
• Enterprise RAG Retrieval - Accuracy, Performance and Cost Efficiency
  • RAG Enterprise Challenge
  • Build a scalable AI factory for RAG
  • RAG Retrieval workflow
  • Key Metrics for RAG Retrieval
  • Target Use cases
  • RAG Balanced Performance
    • Understanding Balanced performance
  • Key Benchmarks for RAG Retrieval
    • Benchmark overview
    • Deep Dive on Benchmark measurement
  • Key Factors affecting Accuracy and Performance
    • Summary for Key Factors
  • Deep Dive into Key Factors affecting Accuracy, Performance and Cost Efficiency
  • Chunk Size Impact on Latency
  • Model and Chunk Size Impact on Throughput (RPS)
  • TopK and Reranker ON/OFF Impact
    • Reranker Impact on Throughput
    • Reranker Impact on Latency
    • VDB TopK Impact on Latency
    • VDB TopK Impact on Output Throughput
    • Latency Benchmark - Usecase Chat 128/128, Reranker ON, Vary TopK
• RAG Scaling Guidelines
  • RAG Baseline Scale 1X
  • RAG Scale nX - Scale 16X example
  • RAG Latency constraints
  • RAG Scaling Efficiency
  • Latency bottlenecks with metrics and tracing
  • RAG Linear scaling ratios and Latency thresholds
  • RAG Linear Scaling Methodology
    • RAG Linear Scaling Ratio summary table
    • RAG Scaling Table Scale 1X to 96X
  • RAG Linear Scaling Example
• RAG Sizing Guidelines
  • RAG Sizing for Chat 128/128
  • RAG Sizing for Summarization (1024/512)
• RAG Benchmarking methodology
  • RAG Benchmarking Test Plan
    • Component scaling
    • VDB TopK and Reranker TopK
  • AIPerf Tool for Benchmarking RAG
• RAG Retrieval Performance Results
  • RAG Chat (128/128) Retrieval Performance
  • RAG Chat Balanced Performance Scale 1X - 96X
  • RAG Chat Time to First Token Latency
    • RAG Chat (128/128) - TTFT - RTX PRO 6000 - Scale 1X - 96X
    • RAG Chat (128/128) - TTFT - H200 NVL - Scale 1X - 20X
  • RAG Chat Latency Constraints
  • RAG Chat Inter-Token Latency
    • RAG Chat (128/128) - ITL - RTX PRO 6000 - Scale 1X - 96X
    • RAG Chat (128/128) - ITL - H200 NVL - Scale 1X - 20X
  • RAG Chat E2E Request Latency
    • RAG Chat (128/128) - E2E Request Latency - RTX PRO 6000 - Scale 1X - 96X
    • RAG Chat (128/128) - E2E Request Latency - H200 NVL - Scale 1X - 20X
  • RAG E2E Latency Degradation with Scale
  • RAG Chat TCO and Scaling Efficiency
    • RAG Chat (128/128) - TCO - RTX PRO 6000 - Scale 1X - 96X
    • RAG Chat (128/128) - TCO - H200 NVL - Scale 1X - 20X
  • RAG Summarization (1024/512) Performance
    • RAG Summarization - TCO - H200 NVL - Scale 1X - 20X
• Summary - Enterprise RAG Retrieval
  • Scaling Best Practices
    • Baseline Configuration - Scale 1X
    • Linear Scaling Ratios
    • RAG Scaling Table - 1X to 96X
    • Scaling Efficiency
    • Balanced Performance
  • Autoscaling with HPA
  • Observability and Monitoring
  • Metrics and Performance Targets
    • Key Performance Targets
    • Latency Constraint Buckets
    • GPU Comparison - RTX PRO 6000 vs H200 NVL
    • RAG ISL - Context Overhead
    • Chunk Size - Select based on use case
    • TopK - Number of Retrieved Chunks
    • Reranker Strategy
  • Sizing Guide - Chat and Summarization
    • Chat Workload: 128 ISL / 128 OSL
    • Summarization Workload: 1024 ISL / 512 OSL
  • RAG Performance - Chat and Summarization
• References

Notices

• Notices
  • NVIDIA Enterprise Reference Architecture Notice & Disclaimers