Enterprise RAG Deployment Guide

NVIDIA Enterprise Reference Architecture

• Abstract
  • Definitions, Acronyms, Abbreviations
• Introduction
  • Scope
  • Enterprise RAG Overview
  • Intended Audience
• Enterprise Reference Architecture Overview
  • Enterprise RA Hardware Stack
  • Enterprise RA Software Stack
• Enterprise RAG on Enterprise RA 285
  • RAG Ingestion Pipeline
    • RAG Ingestion Pipeline Services
      • Ingestion Server (RAG Blueprint)
      • NeMo Retriever (NV-Ingest) Client
      • NeMo Retriever (NV-Ingest) server
      • Nemotron Embedding NIM
      • NeMo Retriever Extraction NIM microservices
      • Vector Database (VDB) - Milvus
      • Enterprise Dataset
  • RAG Retrieval Pipeline
    • RAG Retrieval Pipeline services
      • RAG server
      • LLM NIM
      • Embedding NIM
      • Reranking NIM
      • Vector DB - Milvus
      • Query Rewriter
      • NeMo Guardrails
      • Reflection/Evaluator Agent
      • Inference Benchmarking tool
      • Observability - Metrics, Tracing, Logs
      • Agentic RAG
      • Skills
  • Summary: RAG workload deployment
• Deploy Enterprise RAG on Enterprise RA
  • Pre-requisites to deploy Enterprise RAG
  • Create NGC API secrets on the cluster
  • Deploying NIM microservices for Enterprise RAG
• Deploy the RAG Ingestion Pipeline
  • Configure MIG
    • Step 1: Set GPU Operator MIG strategy
    • Step 2a: Create custom MIG configMap and apply
    • Step 2b: Use a predefined mixed MIG configMap profile
    • Step 3a: Label the node with the custom MIG config label
    • Step 3b: Label the node with the predefined MIG config label
    • Step 4: Remove MIG config on the node
  • Configure Run:ai
    • Step 1: Connect to the Run:ai user interface (UI)
    • Step 2: Label Run:ai worker nodes
    • Step 3: Create a NodePool
    • Step 4: Create a Project
    • Step 5: Create Credentials or user secrets
  • MIG: NeMo Retriever (NV-Ingest) with MIG Profiles
    • Step 1: Create a custom MIG values file
    • Step 2: Deploy Nemo Retriever (NV-Ingest) with Helm
    • Step 3: Verify NIM microservices deployed on MIG
    • Step 4: Scale NIM microservices and NV-Ingest on MIG
    • Limitation: NV-Ingest gRPC reset
  • Run:ai: NeMo Retriever (NV-Ingest) with GPU Fractions
    • Limitation: Run:ai with NIM Operator
    • Step 1: Create a custom Run:ai values file and deploy
    • Step 2: Manually deploy NIM microservices using manifest file
    • Step 3: Verify NIM microservices deployed on Run:ai
    • Step 4: Scale NIM microservices and NV-Ingest on Run:ai
  • Install NV-Ingest client
  • Dataset - Download Enterprise dataset
  • Deploy Vector database (Milvus)
  • Deploy Milvus (Distributed) with GPU index and CPU search using MIG or Run:ai
    • Step 1: Create Helm custom values file
    • Step 2: Deploy Milvus Distributed with Helm
    • Step 3: Verify Milvus Distributed deployment
    • Step 4: View Vector DB status in dashboard or frontend console
    • Step 5: Scaling the Data/Index node
  • Deploy RAG Retrieval pipeline
    • Step 1: Create NIM Cache for Retrieval NIM microservices
    • Step 2: Deploy NIM Service for NIM microservices
    • Step 3: Expose NIM Service with a Load Balancer
    • Step 4: Scaling the NIM Service
  • Deploy RAG Blueprint
    • Step 1: Create the RAG Blueprint custom Helm values file
    • Step 2: Tuning for Performance and Scale
    • Step 3: Deploy RAG blueprint Distributed with Helm
    • Step 4: Verify RAG Blueprint deployment
  • Install AI-Perf benchmarking tool
  • Run AI-Perf Benchmarking tool
• Summary - RAG Deployment Best Practices
  • RAG Retrieval (Blueprint) Deployment
  • LLM NIM Deployment
  • RAG Server Configuration
  • Performance Tuning - Disable Non-Essential Features
  • Observability
  • Vector DB - Milvus Distributed Deployment
• RAG Ingestion Deployment
  • NeMo Retriever (NV-Ingest) Extraction
  • NV-Ingest and Redis Configuration
  • Run:ai Milvus and NIM Deployment
  • Allocating GPU Fractions
• References
• Appendix
  • MIG - NeMo Retriever (NV-Ingest) MIG custom Helm values file
  • Run:ai 1 - NeMo Retriever (NV-Ingest) custom Helm values file
  • Run:ai 2 - NeMo Retriever (NV-Ingest) NIM manifest deployment
  • Milvus distributed Helm custom values file
  • NIM Cache for Retrieval NIM
  • NIM Service for Retrieval NIM microservices
  • RAG Blueprint Helm custom values

Notices

• Notices
  • Notice
  • Trademarks
  • Copyright