RAG Ingestion Deployment#

For Data extraction use NeMo Retriever (NV-Ingest) with GPU accelerated GPU Extraction and Embedding to reduce ingest time from days (using CPU) to hours or minutes.

  • Data extraction is still very CPU intensive; even with some tasks like object detection offloaded to the GPU. To improve GPU utilization efficiency, enable GPU sharing using Run:ai time slicing with GPU fractions or MIG with strict hardware partitioning.

  • Run:ai provides flexibility to allocate GPU fractions (based on memory) for each NIM depending on type data being ingested; table heavy vs image heavy. Run:ai has a custom Kubernetes (KAI) scheduler along with enterprise ready features; bin-packing, workload priorities, preemption policies and fairshare balancing of excess resources.

  • With Run:ai set the GPU fractions based on load and multimodal content type. For example for bo767, allocate OCR 40% GPU, page-elements 30% GPU, table-structure 20% GPU and graphic elements 10% GPU to fit all extraction NIM microservices in a single GPU, as shown in table below. Run:ai reduces the number of GPUs needed by NeMo Retriever (extraction, embedding and Vector DB data node) to 1.75 GPUs (instead of 6 full GPUs).

  • Use MIG GPU partitioning if strict isolation is required. MIG reduces the number of GPUs needed by NeMo Retriever (extraction, embedding and Vector DB data node) to 2 GPUs (instead of 6 full GPUs) see table below for MIG profile allocation with RTX PRO 6000.

NeMo Retriever (NV-Ingest) Extraction#

  • Deploy Standalone: Deploy NV-Ingest as a standalone Helm chart (nemo-microservices/NV-Ingest) for full control over NIM GPU allocation (MIG or Run:ai). The standalone deployment decouples the ingestion pipeline from the RAG blueprint, allowing both to be scaled and upgraded independently.

  • Architecture: The NV-Ingest server acts as the document orchestration layer: it pulls requests from Redis, processes each document in 32 page-chunks using Extraction NIM microservices, then embeds extracted text and sends embeddings to Milvus vector DB for indexing.

  • MIG NV-Ingest Deployment: Deploy NIM microservices, Embedding and Extraction NIM microservices; as part of the Helm chart when using MIG and defined resource requests for the MIG profile.

  • Run:ai NV-Ingest deployment: With Run:ai, due to known NIM Operator limitation, disable NIM deployment with NV-Ingest helm chart and manually deploy NIM microservices using Kubernetes manifests.

NV-Ingest and Redis Configuration#

  • NV-Ingest Workers: Set MAX_INGEST_PROCESS_WORKERS: 32 (default 16) and NV_INGEST_MAX_UTIL: 32 for high-concurrency document ingestion. Allocate 24 vCPU and 48 GiB RAM to prevent CPU throttling.

  • Redis Memory: Increase Redis master and replica memory to 12 GiB (default 6 GiB) when using batch sizes > 100 documents. Use fast block storage (e.g., Longhorn.io with local NVMe) for Redis persistent storage.

  • Ephemeral Storage: Increase ephemeral disk (tmpDirSize) to 64 GiB to store temporary extraction data under high concurrency.

Run:ai Milvus and NIM Deployment#

Helm Deployment for Milvus Vector DB: You can deploy the Milvus datanode using Helm by including a specific request for GPU fraction resources.

Addressing NIM Operator Limitations: Current Helm Charts for NeMo Extraction (NV-Ingest) do not support Run:ai annotations when using the NIM Operator. To resolve this, manually deploy Extraction and Embedding NIM microservices using kubectl apply with tailored Kubernetes manifest files. Ensure these manifests specify runai-scheduler as the schedulerName and define the required gpu-fraction annotations.

Allocating GPU Fractions#

Data content - Adjust GPU fractions based on expected document content: text heavy vs table or chart heavy. Use metrics and tracing to determine latency distribution for ingestion workload and adjust if needed based on metrics.

Data size and SLA Constraints - Scale up GPU fractions (strong scaling) if the data amounts increase or if higher ingestion rates are required. Using strong scaling to scale up fractions to start with a larger baseline will reduce overhead of load balancing for smaller NIM fragments.

RAG Ingestion Baseline (1 GPU) - Use GPU fractions, to consolidate the five Extraction NIM microservices and the Milvus data node onto a single GPU. The GPU fractions were selected for the digital corpora - bo767 dataset (Multimodal dataset 65% text, 25% table, 10% charts) => 767 PDFs; 77K embeddings (pages: 55K, tables: 23K, charts: 8K).

This time-slicing approach enhances efficiency, lowering total consumption from six full GPUs to 1 GPU:

NIM / Service

Run:ai GPU Fraction

vCPU

Memory

llama-nemotron-embed-1b-v2

0.25 (25%) GPU

4 vCPU

8 GiB

nemotron-ocr-v1

0.20 (20%) GPU

3 vCPU

4 GiB

nemotron-page-elements-v3

0.15 (15%) GPU

1 vCPU

2 GiB

nemotron-table-structure-v1

0.10 (10%) GPU

1 vCPU

3.5 GiB

nemotron-graphic-elements-v1

0.05 (5%) GPU

0.5 vCPU

3 GiB

Milvus data node

0.25 (25%) GPU

4 vCPU

8 GiB

RAG Ingestion Baseline (1.75 GPU) [Recommended] - For large amounts of data (10K+ multimodal documents) or faster ingestion rates (>5K docs/s), use strong scaling to scale up the Baseline NIM GPU fractions.

NIM / Service

Run:ai GPU Fraction

vCPU

Memory

llama-nemotron-embed-1b-v2

0.5 (50%) GPU

4 vCPU

8 GiB

nemotron-ocr-v1

0.4 (40%) GPU

3 vCPU

4 GiB

nemotron-page-elements-v3

0.3 (30%) GPU

1 vCPU

2 GiB

nemotron-table-structure-v1

0.2 (20%) GPU

1 vCPU

3.5 GiB

nemotron-graphic-elements-v1

0.1 (10%) GPU

0.5 vCPU

3 GiB

Milvus data node

0.25 (25%) GPU

4 vCPU

8 GiB