Performance and Memory for NVIDIA NIM for Image OCR (NeMo Retriever OCR)

Optimize Image Decoding

For the highest throughput, encode input images as JPEG. JPEG input enables batched image decoding, which is substantially more performant than processing PNG input, especially for requests that contain multiple images. PNG remains supported, but use JPEG for performance-sensitive production workloads and representative throughput benchmarks.

Ensure that the media type in each data URL matches the encoded image bytes. Use data:image/jpeg;base64,... for JPEG input and data:image/png;base64,... for PNG input.

Benchmark Performance

To benchmark NVIDIA NIM for Image OCR (NeMo Retriever OCR), use GenAI-Perf. genai-perf is pre-installed in the Triton Server SDK container. The following examples use GenAI-Perf 0.0.11 from Triton Server SDK 25.02.

Create a JSONL dataset that contains one local image path per line. Use images that represent the documents expected in production. Specify each image path as it appears in the SDK container; GenAI-Perf resolves relative image paths from the current working directory, not from the directory that contains the JSONL file.

{"image": "/datasets/assets/image_01.jpg"}
{"image": "/datasets/assets/image_02.jpg"}
{"image": "/datasets/assets/image_n.jpg"}

Start the Triton Server SDK container and mount the directory that contains images.jsonl and the image assets at /datasets.

export RELEASE="25.02"

docker run -it --rm \
  --gpus=all \
  --network=host \
  --mount type=bind,source=${PWD}/datasets,target=/datasets \
  nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk

In the SDK container, run a latency-oriented benchmark with one image per request and one concurrent request.

genai-perf profile \
  --model nvidia/nemotron-ocr-v2 \
  --service-kind openai \
  --endpoint-type image_retrieval \
  --endpoint v1/ocr \
  --batch-size-image 1 \
  --input-file /datasets/images.jsonl \
  --request-count 300 \
  --concurrency 1 \
  --url http://localhost:8000

For a throughput-oriented benchmark, launch the NIM with NIM_PERFORMANCE_MODE=1, then set --batch-size-image 16 and --concurrency 8. For the multilingual model on NVIDIA L4 and A10G GPUs, also set NIM_ENGINE_COUNT=1; the validated throughput configuration for those GPUs uses one OCR engine.

If requests time out at high concurrency, reduce --concurrency and retry the benchmark. For the complete command-line reference, see the GenAI-Perf command-line options.

Benchmarks

All latency measurements are reported in milliseconds, and throughput is reported in images per second. P50 latency is request_latency.p50 from the latency-english or latency-multilingual configuration with image batch size 1 and concurrency 1. Throughput is image_throughput.avg from the applicable throughput configuration with image batch size 16 and concurrency 8.

English Model

Latency

GPU	P50 Latency (ms)
NVIDIA A100 SXM4 80GB	18.7
NVIDIA A10G	39.6
NVIDIA GB200	10.3
NVIDIA H100 80GB HBM3	12.4
NVIDIA L40S	15.0
NVIDIA RTX PRO 6000 Blackwell Server Edition	10.7

Throughput

GPU	Throughput (images/s)
NVIDIA A100 SXM4 80GB	67.7
NVIDIA A10G	28.0
NVIDIA GB200	147.8
NVIDIA H100 80GB HBM3	127.8
NVIDIA L40S	70.4
NVIDIA RTX PRO 6000 Blackwell Server Edition	129.9

Multilingual Model

Latency

GPU	P50 Latency (ms)
NVIDIA A100 SXM4 80GB	20.8
NVIDIA A10G	48.9
NVIDIA GB200	10.5
NVIDIA H100 80GB HBM3	13.4
NVIDIA L40S	16.8
NVIDIA RTX PRO 6000 Blackwell Server Edition	11.7

Throughput

GPU	Throughput (images/s)
NVIDIA A100 SXM4 80GB	52.9
NVIDIA A10G	19.7
NVIDIA GB200	129.0
NVIDIA H100 80GB HBM3	102.1
NVIDIA L40S	54.6
NVIDIA RTX PRO 6000 Blackwell Server Edition	100.3

The multilingual throughput result for NVIDIA A10G uses the throughput-multilingual-small configuration with NIM_ENGINE_COUNT=1. The other multilingual throughput results use throughput-multilingual, and all English throughput results use throughput-english.

Performance varies with document content, image resolution, OCR model variant, batch size, concurrency, and runtime configuration. Benchmark with representative production inputs before sizing a deployment.

Memory Footprint

Where available, the approximate GPU memory sizes in the following table are the startup measurements from the latency configuration after forward buffers are allocated. Runtime GPU memory use can increase with larger batches, additional OCR engines, CUDA graph capture shapes, and input-dependent working buffers.

GPU	English approximate GPU memory size (GiB)	Multilingual approximate GPU memory size (GiB)
NVIDIA A100 SXM4 80GB	1.79	2.88
NVIDIA A10G	1.63	2.72
NVIDIA B200	2.17	3.26
NVIDIA GB10 Grace Blackwell Superchip, such as NVIDIA DGX Spark systems	N/A	N/A
NVIDIA GB200	2.23	3.33
NVIDIA H100 80GB HBM3	2.07	3.17
NVIDIA H100 NVL	2.07	3.17
NVIDIA H200	2.07	3.17
NVIDIA L4	1.57	2.66
NVIDIA L40S	1.80	2.89
NVIDIA RTX PRO 6000 Blackwell Server Edition	1.94	3.03

Note

GB10 uses unified system memory. GPU memory usage is reported as N/A because the usual VRAM measurements are not accurate for this architecture. For more information, refer to Guidance for Reporting Memory Resources with Unified Memory Architecture.

Throughput mode defaults to two OCR engines and a batch size of 16, which requires more GPU memory than latency mode.

For the GPU SKUs and precisions that the NIM supports, refer to Support Matrix for NVIDIA NIM for Image OCR (NeMo Retriever OCR).