Is this page helpful?

Optimization for NVIDIA NIM for Image OCR (NeMo Retriever OCR)#

Use this documentation to optimize NVIDIA NIM for Image OCR (NeMo Retriever OCR).

Model Profiles and Kernel Selection#

Starting in version 2.0, NeMo Retriever OCR uses a purpose-built CUDA inference runtime and has no notion of model profile selection. No profile selection step is required, and the list-model-profiles command is not available.

NeMo Retriever OCR supports FP16 only. Depending on the GPU in your deployment, the NIM might use performance-tuned kernels. The runtime determines this internally, you can’t control it, and the behavior can change between releases.

Select Latency or Throughput Mode#

The default configuration is optimized for request latency:

-e NIM_PERFORMANCE_MODE=0

For higher image throughput, enable throughput mode:

-e NIM_PERFORMANCE_MODE=1

Throughput mode changes the defaults to two OCR engines, a maximum batch size of 16 images, a 10-millisecond batching wait, and throughput-oriented server scheduling. Explicit values for NIM_ENGINE_COUNT, NIM_PIPELINE_MAX_BATCH_SIZE, NIM_SERVER_MAX_WAIT_MS, or NIM_SERVER_MODE take precedence over the mode defaults.

Increasing the engine count or maximum batch size can increase VRAM use. If the NIM fails to start with an out-of-memory error, use latency mode or explicitly set NIM_ENGINE_COUNT=1 and NIM_PIPELINE_MAX_BATCH_SIZE=1.

Select the OCR Model Variant#

The English and multilingual models have different performance characteristics. If the corpus being processed contains only English text, 20-30% higher throughput can be achieved by deploying the English model.

The multilingual model is served by default. To serve the English-only model, set the following variable when you launch the container:

-e NIM_ENGINE_MODEL_VARIANT=english

To select the multilingual model explicitly, set NIM_ENGINE_MODEL_VARIANT=multilingual.

Configure CUDA Graph Capture and Warmup#

The runtime enables full detector CUDA graph capture and warmup by default. Most deployments should keep the defaults. Advanced deployments can tune detector, recognizer, and relational-stage capture or warmup independently by using the NIM_ENGINE_OCR_* environment variables.

Capture and warmup shapes must reflect the batch and region-chunk sizes used by the deployment. Expanding the captured shapes can improve steady-state performance but increases startup time and VRAM use. For the complete list of controls, refer to Environment Variables for NVIDIA NIM for Image OCR (NeMo Retriever OCR).

Override Profile Selection#

Starting in version 2.0, there is no model profile workflow, and the NIM_MODEL_PROFILE environment variable is not supported.