Performance for NVIDIA NIM for Image OCR (NeMo Retriever OCR)

Optimize Image Decoding

For the highest throughput, encode input images as JPEG. JPEG input enables batched image decoding, which is substantially more performant than processing PNG input, especially for requests that contain multiple images. PNG remains supported, but use JPEG for performance-sensitive production workloads and representative throughput benchmarks.

Ensure that the media type in each data URL matches the encoded image bytes. Use data:image/jpeg;base64,... for JPEG input and data:image/png;base64,... for PNG input.

Benchmark Performance

To benchmark NVIDIA NIM for Image OCR (NeMo Retriever OCR), use GenAI-Perf. genai-perf is pre-installed in the Triton Server SDK container. The following examples use GenAI-Perf 0.0.11 from Triton Server SDK 25.02.

Create a JSONL dataset that contains one local image path per line. Use images that represent the documents expected in production. Specify each image path as it appears in the SDK container; GenAI-Perf resolves relative image paths from the current working directory, not from the directory that contains the JSONL file.

{"image": "/datasets/assets/image_01.jpg"}
{"image": "/datasets/assets/image_02.jpg"}
{"image": "/datasets/assets/image_n.jpg"}

Start the Triton Server SDK container and mount the directory that contains images.jsonl and the image assets at /datasets.

export RELEASE="25.02"

docker run -it --rm \
  --gpus=all \
  --network=host \
  --mount type=bind,source=${PWD}/datasets,target=/datasets \
  nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk

In the SDK container, run a latency-oriented benchmark with one image per request and one concurrent request.

genai-perf profile \
  --model nvidia/nemotron-ocr-v2 \
  --service-kind openai \
  --endpoint-type image_retrieval \
  --endpoint v1/ocr \
  --batch-size-image 1 \
  --input-file /datasets/images.jsonl \
  --request-count 300 \
  --concurrency 1 \
  --url http://localhost:8000

For a throughput-oriented benchmark, launch the NIM with NIM_PERFORMANCE_MODE=1, then set --batch-size-image 16 and --concurrency 8. For the multilingual model on NVIDIA L4 and A10G GPUs, also set NIM_ENGINE_COUNT=1; the validated throughput configuration for those GPUs uses one OCR engine.

If requests time out at high concurrency, reduce --concurrency and retry the benchmark. For the complete command-line reference, see the GenAI-Perf command-line options.

Benchmarks

All latency measurements are reported in milliseconds, and throughput is reported in images per second. P50 latency is request_latency.p50 from the latency-english or latency-multilingual configuration with image batch size 1 and concurrency 1. Throughput is image_throughput.avg from the applicable throughput configuration with image batch size 16 and concurrency 8.

English Model

Latency

GPU SKU	P50 Latency (ms)
NVIDIA-A100-SXM4-80GB	18.7
NVIDIA-A10G	39.6
NVIDIA-GB200	10.3
NVIDIA-H100-80GB-HBM3	12.4
NVIDIA-L40S	15.0
NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition	10.7

Throughput

GPU SKU	Throughput (images/s)
NVIDIA-A100-SXM4-80GB	67.7
NVIDIA-A10G	28.0
NVIDIA-GB200	147.8
NVIDIA-H100-80GB-HBM3	127.8
NVIDIA-L40S	70.4
NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition	129.9

Multilingual Model

Latency

GPU SKU	P50 Latency (ms)
NVIDIA-A100-SXM4-80GB	20.8
NVIDIA-A10G	48.9
NVIDIA-GB200	10.5
NVIDIA-H100-80GB-HBM3	13.4
NVIDIA-L40S	16.8
NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition	11.7

Throughput

GPU SKU	Throughput (images/s)
NVIDIA-A100-SXM4-80GB	52.9
NVIDIA-A10G	19.7
NVIDIA-GB200	129.0
NVIDIA-H100-80GB-HBM3	102.1
NVIDIA-L40S	54.6
NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition	100.3

The multilingual throughput result for NVIDIA-A10G uses the throughput-multilingual-small configuration with NIM_ENGINE_COUNT=1. The other multilingual throughput results use throughput-multilingual, and all English throughput results use throughput-english.

Performance varies with document content, image resolution, OCR model variant, batch size, concurrency, and runtime configuration. Benchmark with representative production inputs before sizing a deployment.