Optimization for NVIDIA NIM for Nemotron Parse#
Use this documentation to optimize NVIDIA NIM for Nemotron Parse.
Model Profiles and Kernel Selection#
Parse NIM uses a purpose-built CUDA inference runtime and has no notion of model profile selection.
No profile selection step is required, and the list-model-profiles command is not available.
The NIM_MODEL_PROFILE environment variable is not supported.
Nemotron Parse NIM runs in BF16 on all supported GPUs, and BF16 is optimized. For details, refer to Support Matrix for NVIDIA NIM for Nemotron Parse.
Depending on the GPU in your deployment, the NIM might use performance-tuned kernels. The runtime determines this internally, you can’t control it, and the behavior can change between releases.
Increase Throughput with Decoder Capacity#
By default, the NIM runs with one decoder slot. To process more requests in parallel, increase the decoder capacity:
-e NIM_PIPELINE_MAX_BATCH_SIZE=8
Higher capacity uses more GPU memory. If the NIM fails with an out-of-memory error, reduce NIM_PIPELINE_MAX_BATCH_SIZE. The supported range is 1 through 256. Each request still contains one image.
When decoder capacity is unset, NIM_PERFORMANCE_MODE=1 selects 128 slots; an explicit NIM_PIPELINE_MAX_BATCH_SIZE takes precedence. For queue limits, fail-fast admission, and timeout behavior, refer to Environment Variables for NVIDIA NIM for Nemotron Parse.
Limit the Output Length#
Each request generates up to options.decoding.max_output_tokens tokens, with a default and maximum of 9,000. Dense pages, such as pages with large tables or newspaper-style layouts, need the full budget. If your pages are known to be simple, a lower limit can reduce generation work. A lower limit also increases the chance that the response ends before the page is completely represented, which the NIM reports with finish_reason: "length". For details, refer to API Usage Examples (OpenAPI) for NVIDIA NIM for Nemotron Parse.