Optimization for NVIDIA NIM for Image OCR (NeMo Retriever OCR)#
Use this documentation to optimize NVIDIA NIM for Image OCR (NeMo Retriever OCR).
Starting in release 2.0.0, the NIM uses a purpose-built CUDA inference runtime. The NIM does not use Triton model profiles, and the list-model-profiles command and NIM_MODEL_PROFILE environment variable are not available.
Select Latency or Throughput Mode#
The default configuration is optimized for request latency:
-e NIM_PERFORMANCE_MODE=0
For higher image throughput, enable throughput mode:
-e NIM_PERFORMANCE_MODE=1
Throughput mode changes the defaults to two OCR engines, a maximum batch size of 16 images, a 10-millisecond batching wait, and throughput-oriented server scheduling. Explicit values for NIM_ENGINE_COUNT, NIM_PIPELINE_MAX_BATCH_SIZE, NIM_SERVER_MAX_WAIT_MS, or NIM_SERVER_MODE take precedence over the mode defaults.
Increasing the engine count or maximum batch size can increase VRAM use. If the NIM fails to start with an out-of-memory error, use latency mode or explicitly set NIM_ENGINE_COUNT=1 and NIM_PIPELINE_MAX_BATCH_SIZE=1.
Select the OCR Model Variant#
The English and multilingual models have different performance characteristics. If the corpus being processed contains only English text, 20-30% higher throughput can be achieved by deploying the English model.
The multilingual model is served by default. To serve the English-only model, set the following variable when you launch the container:
-e NIM_ENGINE_MODEL_VARIANT=english
To select the multilingual model explicitly, set NIM_ENGINE_MODEL_VARIANT=multilingual.
Configure CUDA Graph Capture and Warmup#
The runtime enables full detector CUDA graph capture and warmup by default. Most deployments should keep the defaults. Advanced deployments can tune detector, recognizer, and relational-stage capture or warmup independently by using the NIM_ENGINE_OCR_* environment variables.
Capture and warmup shapes must reflect the batch and region-chunk sizes used by the deployment. Expanding the captured shapes can improve steady-state performance but increases startup time and VRAM use. For the complete list of controls, see Environment Variables for NVIDIA NIM for Image OCR (NeMo Retriever OCR).