RAG Ingestion Sizing

In order to size the Enterprise RAG ingestion, estimate the Peak Usage in terms of total amount of data, maximum number of documents that need to be ingested at a given time. In addition, determine the ingestion rate required, in terms of docs/s or vectors/s, that need to be supported for each data type: Multimodal PDF, Text PDF or Text parquet.

For a given enterprise organization size, what is the peak number of documents to be ingested at any given time. Start conservatively and then refine the estimate; scale when higher data ingest rates are needed, add more servers and GPUs over time.

The sizing table below provides sizing recommendations for multimodal PDF documents using data from Multimodal Ingestion Scaling with Run:ai and MIG.

Enterprise document dataset bo767 documents average pages: 71 pages/doc, average tables: 30 tables/doc, charts: 10 charts/doc.

Embedding NIM dimensions: 2048 and quantization: FP32.

RAG Ingestion T-Shirt Sizing with Run:ai and MIG

TCO is calculated using ERA cluster - RTX PRO 6000 GPU hourly cost: $0.97/hour.

Type	Ingest rate	Scale Recommendation	Extract + Embed time	VDB ingest time	TCO Total Cost
MM-Small-5K — Multimodal Small — 5K docs, 500K vectors	1.5K docs/hr — 150K vectors/hour	Run:ai Scale-1X — GPU:1 — Concurrency: 16	3.10 hours — GPU:0.75	0.20 hours — GPU:0.25	$2.59
MM-Small-5K — Multimodal Small — 5K docs, 500K vectors	1.5K docs/hr — 150K vectors/hour	MIG Scale-1X — GPUs:1.5 — Concurrency: 16	3.10 hours — GPU:0.75	0.20 hours — GPU:0.25	$4.28
MM-Medium-10K — Multimodal Medium — 10K docs, 1M vectors	2.7K docs/hour — 270K vectors/hour	Run:ai Scale 2X — GPUs:1.75 — Concurrency: 32	3.48 hours — GPU:1.5	0.20 hours — GPU:0.25	$5.47
MM-Medium-10K — Multimodal Medium — 10K docs, 1M vectors	2.7K docs/hour — 270K vectors/hour	MIG-Scale 2X — GPUs:2 — Concurrency: 32	3.5 hours — GPU:1.75	0.20 hours — GPU:0.25	$6.39
MM-Large-40K — Multimodal Large — 40K docs, 4M vectors	4.8K docs/hour — 480K vectors/hour	Run:ai Scale 4X — GPUs:3.25 — Concurrency: 64	8.16 hours — GPU:3	0.20 hours — GPU:0.25	$22.50
MM-Large-40K — Multimodal Large — 40K docs, 4M vectors	5K docs/hour — 500K vectors/hour	MIG-Scale 4X — GPUs:4 — Concurrency: 64	7.68 hours	—	$25.23
MM-X-Large-100K — Multimodal X-Large — 100K docs, 10M vectors	7K docs/hour — 700K vectors/hour	Run:ai-Scale 6X — GPUs:4.85 — Concurrency: 96	14.29 hours	—	$52.51
MM-X-Large-100K — Multimodal X-Large — 100K docs, 10M vectors	6.5K docs/hour — 650K vectors/hour	MIG-Scale 6X — GPUs:5.5 — Concurrency: 96	15.35 hours	—	$65.84

Multimodal Small (MM-Small-5K) is the small data size for ingesting up to 5K documents at a time, which generates up to ~500K vector embeddings (2048 dim X FP32) and requires an ingestion rate of at least 1,500 documents per hour and 150K vectors per hour.

The recommendation is to use Scale 1X with at least 16 concurrent document requests using either Run:ai (1 GPU) or MIG (1.5 GPUs). E2E ingest time is 3.30 hours.

E2E ingest time =  Embed time (includes extraction)  +  Vector DB ingest time
                 =  3.10 hours (3 hours 6 min) + 0.20 hours (12 min)
                 =  3.3 hours

TCO Total cost to ingest 5K docs, 500K vectors: Run:ai is $2.69 and MIG is $4.28, given the RTX PRO 6000 GPU hourly cost: $1.09/hour.

TCO Total Cost =  Embed cost (includes extraction)  +  Vector DB cost
                 =  GPU_hourly_cost X (embed_gpus X embed_time_hours +
                                       vdb_gpus X vdb_time_hours)
 Run:ai TCO = $ 1.09 X ( 0.75 GPUs X 3.10 hours + 0.25 GPUs X 0.20 hours)
 MIG    TCO = $ 1.09 X ( 1.25 GPUs X 3.10 hours + 0.25 GPUs X 0.20 hours)

Ingestion Sizing — T-Shirt Sizing Guide (Extended)

Use the Sizing table below to size GPU resources for multimodal PDF ingestion. This is based on Nemo Retriever (NV-Ingest) multimodal benchmarks with the bo767 dataset (average 71 pages/doc, 30 tables/doc, 10 charts/doc).

Assuming Enterprise RA 2-8-5 cluster RTX PRO 6000 GPU hourly cost: $0.97/hour (including software and server, 5-year amortization).

Size	Data Volume	Required Rate	Run:ai Config	MIG Config	E2E Time	Run:ai TCO	MIG TCO
Small	5K docs — 500K vectors	1,500 docs/hr — 150K vector/hr	Scale 1X — 1 GPU — CR=16	Scale 1X — 1.5 GPUs — CR=16	3.30 hrs	$2.59	$4.28
Medium	10K docs — 1M vectors	2,700 docs/hr — 270K vector/hr	Scale 2X — 1.75 GPUs — CR=32	Scale 2X — 2 GPUs — CR=32	3.68 hrs	$5.47	$6.39
Large	40K docs — 4M vectors	4,800 docs/hr — 480K vector/hr	Scale 4X — 3.25 GPUs — CR=64	Scale 4X — 4 GPUs — CR=64	8.36 hrs	$22.50	$25.23
X-Large	100K docs — 10M vectors	7,000 docs/hr — 700K vector/hr	Scale 6X — 4.85 GPUs — CR=96	Scale 6X — 5.5 GPUs — CR=96	14.29 hrs	$52.51	$65.84

Start conservatively with Scale 1X or 2X and refine as actual enterprise document profiles (pages, tables, charts) become known. The biggest uncertainty is multimodal extraction time, which varies significantly with document content type.

Sizing considerations:

• Data Volume - How many documents need to be ingested at a given time, pages per document, document size MB? What is the type of content, Text, PDF, parquet? What is modality percentage, text vs tables/charts/image?

• Ingestion rate - What is the document or vector ingestion rate required, in terms of docs/s or vectors/s? Use benchmarking data to determine the document ingestion rate possible at different concurrency for each type: Multimodal PDF, Text or parquet.

• Ingestion schedule - Is ingestion continuous, GPUs permanently allocated? or Does ingestion need to be scheduled during specific off-peak hours, nights or weekends?

Based on data type, volume and schedule; select the sizing (small, medium, large) to achieve the ingestion rate required.

• TCO Total Cost - Calculate the TCO Cost using formula below:

TCO Cost per Million vectors = GPU_hourly_cost × (1M vectors / 3600s)
               ((embed_gpus / embed_rate) + (vdb_gpus / vdb_rate))

Small Scale-1X Run:ai TCO: $1.09 × (0.75 GPU × 3.10 hr + 0.25 GPU × 0.20 hr) = $2.59