Is this page helpful?

Optimization for NVIDIA NIM for Object Detection#

Use this documentation to learn about optimization for NVIDIA NIM for Object Detection.

Model Profiles and Kernel Selection#

Starting in version 2.0, Object Detection NIM has no notion of model profile selection. No profile selection step is required, and the list-model-profiles command is not available.

Object Detection NIM supports FP16 only. Depending on the GPU in your deployment, the NIM might use performance-tuned kernels. The runtime determines this internally, you can’t control it, and the behavior can change between releases.

Precision#

Object Detection NIM supports only FP16.

Leave NIM_ENGINE_PRECISION unset or set it to fp16.

docker run ... \
  -e NIM_ENGINE_PRECISION=fp16 \
  $IMG_NAME

Latency and Throughput#

Use NIM_PERFORMANCE_MODE to select latency-oriented or throughput-oriented defaults for unset variables.

Scenario

Recommended Settings

Lower latency, single-client workloads

NIM_PERFORMANCE_MODE=0, NIM_SERVER_MODE=latency, NIM_SERVER_MAX_WAIT_MS=0, NIM_ENGINE_COUNT=1, NIM_PIPELINE_MAX_BATCH_SIZE=1

Higher throughput, sustained concurrent workloads

NIM_PERFORMANCE_MODE=1, NIM_SERVER_MODE=throughput, NIM_SERVER_MAX_WAIT_MS=10, NIM_ENGINE_COUNT=2, NIM_PIPELINE_MAX_BATCH_SIZE=16

You can set any of the controlled variables explicitly to override the NIM_PERFORMANCE_MODE default.

Warmup and Runtime Artifact Cache#

The object detection runtime can compile and cache runtime artifacts under NIM_ENGINE_PRECOMPILE_CACHE_DIR. When unset, mount /opt/cache and let the container use the default cache location.

To precompile cuDNN plans and related runtime artifacts and then exit before the server starts, run the container with NIM_ENGINE_PRECOMPILE_ONLY=1.

docker run -it --rm \
  --runtime=nvidia \
  --gpus '"device=0"' \
  -e HF_TOKEN \
  -e NIM_ENGINE_MODEL_NAME="nvidia/nemotron-page-elements-v3" \
  -e NIM_ENGINE_MODEL_PATH="/model/page-elements" \
  -e NIM_ENGINE_WARMUP_MODE=full \
  -e NIM_ENGINE_PRECOMPILE_ONLY=1 \
  -v "$LOCAL_NIM_CACHE/cache:/opt/cache" \
  -v "$LOCAL_NIM_CACHE/weights:/model" \
  $IMG_NAME

NIM_ENGINE_WARMUP_MODE, NIM_ENGINE_WARMUP_SHAPES, NIM_ENGINE_CUDA_GRAPH_CAPTURE, and NIM_ENGINE_CUDA_GRAPH_CAPTURE_SHAPES provide advanced control over warmup and CUDA graph capture.

Workload

Recommended Settings

Variable batch sizes in production

NIM_ENGINE_WARMUP_MODE=full, NIM_ENGINE_CUDA_GRAPH_CAPTURE=full

Fixed batch size known ahead of time

NIM_ENGINE_WARMUP_MODE=partial, NIM_ENGINE_CUDA_GRAPH_CAPTURE_SHAPES=<batch-size>

Development, CI, or ephemeral startup

NIM_ENGINE_WARMUP_MODE=none

NIM_ENGINE_WARMUP_MODE=none starts faster, but the first requests for each batch size can be slower while cuDNN plans are selected inline. Use NIM_ENGINE_WARMUP_MODE=full for stable performance benchmarks and long-running production deployments with varied batch sizes.

NIM_PIPELINE_MAX_BATCH_SIZE is the GPU memory lever. Changing warmup or graph capture scope does not reduce the preallocated tensor buffer size.

Override Profile Selection#

Starting in version 2.0, there is no model profile workflow, and the NIM_MODEL_PROFILE environment variable is not supported.