Abstract

As enterprises integrate AI Inference solutions into their business workflows (IT tech support, Customer service, engineering code analysis), they are looking to bring their own business data sources (blogs, product datasheets, sales reports, bugs, logs, meeting notes) and have the AI workloads use their enterprise data to provide relevant and accurate responses with reduced hallucinations. This is where RAG (Retrieval-augmented generation) can be leveraged to provide context aware AI Inference with enterprise business data on NVIDIA Enterprise Reference Architecture.

Typical Enterprise use cases for query-response AI Inference are listed below

• IT or Customer support AI chatbot with single or multi-turn conversation on new product features using data from product datasheets, blogs, performance test results, roadmaps, internal bug reports and system logs.

• Summarization and overview of customer meetings, internal presentations; overview of research papers or financial report includes multimodal data including tables, charts, images, audio, video

• Generation or Analysis - code generation and research from multiple sources; internal and the Internet and creating detailed research reports

  • Financial analysis and forecasting based on company reports (Cash flow, sales reports, bookings, orders)

  • Engineering root cause analysis (RCA) to pinpoint issues by reviewing logs, traces, metrics for multiple components.

  • Code analysis or code generation by reviewing existing source code

This guide provides deployment best practices and recommendations for Enterprise RAG workloads based on NVIDIA RAG Blueprint (Github) on the NVIDIA Enterprise ReferenceArchitectureERA 2-8-5-200 RTX PRO 6000 and H200 NVL.

This is the first in a series of three documents covering Enterprise RAG deployment, RAG ingestion scaling and RAG Retrieval Scaling and Sizing:

• Enterprise RAG Deployment Guide v2.0

• Enterprise RAG Ingestion Scaling Guide v2.0

• Enterprise RAG Retrieval Scaling and Sizing Guide v2.0

Note

For the detailed hardware design, refer to NVIDIA Reference Architecture for AI Factory with RTX PRO 2-8-5-200.

Definitions, Acronyms, Abbreviations

Term	Definition
NGC	NVIDIA NGC™ is the portal of enterprise services, software, management tools, and support for end-to-end AI, MLOPs and HPC workflows. NGC is the repository or container catalog for NVIDIA optimized models, NeMo Retriever microservices, NIM microservices, NVIDIA Kubernetes operators (Network operator, GPU Operator, NIM Operator), Helm charts.
LLM	Large Language Models or LLMs are deep learning algorithms, specifically a type of neural network called a transformer model, designed to understand, generate, and predict human language. They are trained on massive datasets of text, learning to predict the next word or sequence of words in a sentence, based on the context provided.
NVIDIA NIM™	NVIDIA Inference Microservice or NIM is a containerized microservice that supports AI optimized models from NVIDIA and partners that can be deployed and managed using the NIM operator in a Kubernetes cluster or in NVCF.
NVIDIA Blueprint™	NVIDIA Blueprints™ bring together multiple NVIDIA NIM microservices™ and third party microservices (Vector Database, LLM, VLM, AI Agents) to create an end-to-end solution that is fully customizable and easy to deploy on both workstations for AI developers using docker compose and on Enterprise Kubernetes clusters for AI Application teams. NVIDIA Blueprints and NIM microservices are part of NVIDIA AI Enterprise and available for download on the NGC Catalog.
Kubernetes or K8s	Kubernetes is an open source (OSS) Container orchestration platform to deploy and scale microservices. Kubernetes is used in most enterprise data centers to deploy AI workloads at scale for enterprise users and customers.
NVIDIA AI Enterprise Software	NVIDIA AI Enterprise includes the foundation blocks for the GenAI workloads; requires an Entitlement and can be downloaded from the NGC container catalog. It includes Kubernetes operators for GPU, Network and NIM microservices as well as Nvidia accelerated NIM microservices for Blueprint to get started and deploy and end-to-end validated AI inference, training or MLOps stack.
Run:ai	NVIDIA Run:ai is a GPU orchestration and optimization platform that helps organizations maximize compute utilization for AI workloads. Built on Kubernetes, NVIDIA Run:ai supports dynamic GPU allocation, workload submission, workload scheduling, and resource sharing. Run:ai uses a custom Kubernetes scheduler that uses time slicing to enable running AI workloads on GPU fractions.
Multi-instance GPU (MIG)	Multi-Instance GPU (MIG) feature enables secure hardware partitioning of GPU compute and memory into multiple GPU Instances for AI workloads, optimizing GPU utilization. The number of MIG instances depends on GPU type and the supported MIG profiles.
Tokens, Tokenization	A token is a fundamental unit of data that is processed by an AI inference model or LLM. During AI inference, the LLM receives a user input prompt — which can be text, image, audio clip, video, sensor data or even gene sequence — that it translates into a series of tokens using a tokenizer and the process is called tokenization. The LLM then processes these input tokens, generates its response as tokens and then translates it to the user’s expected format; language of choice. For the English language, an AI token typically represents about 4 characters or roughly three-quarters (¾) of a word. Tokens are used to define a model’s performance or throughput in terms of output tokens per second. Tokens can also be used to define user experience in terms of average response time per user or tokens per second per user. Tokens drive AI service cost and revenue; pricing plans with limits on number of tokens per minute shared between input and output. Chain of thought reasoning models like Deepseek R1 and Nemotron Llama Ultra; can generate a lot of reasoning tokens that show the thought process before they provide the final answer. Reasoning models can generate 10x - 20x more tokens compared with traditional models with one-shot inference.
AI Performance KPI	AI Performance KPIs (Key Performance Indicators) are metrics used to benchmark AI agents or LLM. These are categorized in Latency and Throughput metrics. Metrics are covered in more detail later in this document: - First Token Latency (FTL) or Time to First Token (TTFT) - Inter Token Latency (ITL) - E2E Request Latency (e2e Latency) - Output Throughput (TPS) - User Throughput (TPS)
ISL, OSL, Context Length	Input Sequence Length (ISL) and Output Sequence Length (OSL) are the number of tokens consumed and generated (respectively) by the LLM. During AI inference, the LLM receives a user input prompt which when converted into tokens; this is the prompt ISL (token count). The LLM then processes these input tokens and generates its response as tokens which gives the response OSL (token count). The total number of tokens that are consumed and generated by the LLM including ISL, OSL and any system prompts that are required to process a single AI inference request is called the Context Length or Context Sequence Length. LLMs typically have maximum OSL and maximum context length. As ISL increases in size, the response time or time to first token (TTFT) also called first token latency (FTL) increases exponentially.
Enterprise RAG	NVIDIA Enterprise RAG (Retrieval-augmented generation) solution is based on the RAG Blueprint and provides context aware AI Inference for query response use cases using enterprise business data. The response from the RAG server is accurate, grounded and relevant to the enterprise user. The Enterprise RAG solution can be reliably deployed on a production Kubernetes cluster based on the Enterprise RA software stack and can scale from 1SU (4 nodes, 32 GPUs) to 4 SU (16 nodes, 128 GPUs) in the NVIDIA RA RTX PRO 285-200 hardware cluster.