Performance and Memory for NVIDIA NIM for Nemotron Parse

Benchmark Performance

For the following measurements, the benchmark client sends one page image per multipart POST /v1/parse request and measures the time until the complete response is received. For the request format, refer to API Usage Examples (OpenAPI) for NVIDIA NIM for Nemotron Parse.

To benchmark your deployment, use a client that supports multipart requests, representative page images, and the concurrency expected in production. Record the output token limit and decoder capacity alongside the latency and throughput results.

About the Measurements

Choose between two recommended NIM configurations based on the GPU memory available:

• Default — Use NIM_PIPELINE_MAX_BATCH_SIZE=1 for lower GPU memory use. This configuration is better for resource-constrained environments and GPUs shared with other workloads.

• Maximum throughput — Use NIM_PIPELINE_MAX_BATCH_SIZE=128, or 64 on A10G, when more GPU memory can be allocated to the NIM. Achieving the reported throughput requires sending enough page images concurrently to keep the NIM’s continuous batching fully utilized.

Minimum-VRAM measurements use the default configuration and one concurrent request. Performance benchmarks cover both configurations: one concurrent request in the default configuration, and 128 concurrent requests, or 64 on A10G, in the maximum-throughput configuration. Both configurations use one page image per request. The memory table reports GPU memory after startup and warmup; memory usage during inference can be higher.

Latency is measured at the benchmark client and includes time waiting for the NIM to process the request. P50 is the 50th percentile of request latency and is reported in seconds. Throughput is the average number of completed page requests per second. Latency and throughput are measured at the concurrency shown in the performance table. Max batch size is the decoder capacity configured by NIM_PIPELINE_MAX_BATCH_SIZE.

Parse latency depends heavily on the amount of content on each page, because the NIM generates up to max_output_tokens tokens for each page. Performance varies with document content, image resolution, the output token limit, decoder capacity (NIM_PIPELINE_MAX_BATCH_SIZE), and concurrency. Benchmark with representative production inputs before sizing a deployment.

For the GPU SKUs and precisions that the NIM supports, refer to Support Matrix for NVIDIA NIM for Nemotron Parse.

Nemotron 3.5 Parse

Memory Footprint

The following table contains the memory footprint data for Nemotron 3.5 Parse at each measured maximum batch size (NIM_PIPELINE_MAX_BATCH_SIZE).

GPU	Precision	Max Batch Size	Approximate GPU Memory Size (GiB)
NVIDIA RTX PRO 6000 Blackwell Server Edition	bf16	1	4.72
NVIDIA RTX PRO 6000 Blackwell Server Edition	bf16	128	34.85
NVIDIA GB200	bf16	1	5.18
NVIDIA GB200	bf16	128	35.3
NVIDIA H100 NVL	bf16	1	5.02
NVIDIA H100 NVL	bf16	128	35.15
NVIDIA A100 SXM4 80GB	bf16	1	4.47
NVIDIA A100 SXM4 80GB	bf16	128	34.59
NVIDIA L40S	bf16	1	4.47
NVIDIA L40S	bf16	128	34.6
NVIDIA A10G	bf16	1	4.3
NVIDIA A10G	bf16	64	19.37

Performance

ParseBench

The performance benchmarks use the following settings:

Setting	Value
Model	Nemotron 3.5 Parse, BF16
GPU count	1 per NIM instance
Dataset	ParseBench, 2,078 page images
Measured requests	2,078
Warmup requests	16, before measurement
Images per request	1
Concurrent requests	Default: 1; maximum throughput: 128 (A10G: 64)
Decoder capacity (NIM_PIPELINE_MAX_BATCH_SIZE)	Default: 1; maximum throughput: 128 (A10G: 64)
Output token limit (options.decoding.max_output_tokens)	8,744
Request timeout	1,800 seconds

The following table contains the ParseBench performance data for Nemotron 3.5 Parse in the default and maximum-throughput configurations.

GPU	Max Batch Size	Concurrency	P50 Latency (s)	Throughput (pages/s)
NVIDIA A100 SXM4 80GB	1	1	0.69	1.08
NVIDIA A100 SXM4 80GB	128	128	14.41	6.56
NVIDIA A10G	1	1	1.32	0.57
NVIDIA A10G	64	64	21.48	2.24
NVIDIA GB200	1	1	0.34	2.17
NVIDIA GB200	128	128	3.92	25.04
NVIDIA H100 NVL	1	1	0.47	1.58
NVIDIA H100 NVL	128	128	7.61	12.44
NVIDIA L40S	1	1	0.68	1.13
NVIDIA L40S	128	128	17.45	5.34
NVIDIA RTX PRO 6000 Blackwell Server Edition	1	1	0.46	1.65
NVIDIA RTX PRO 6000 Blackwell Server Edition	128	128	9.12	10.25