Is this page helpful?

Optimization for NVIDIA NIM for Object Detection#

Use this documentation to learn about optimization for NVIDIA NIM for Object Detection.

Runtime Pipeline Selection#

Object Detection NIM uses a runtime that selects the supported object detection pipeline internally for the loaded model and detected GPU. The list-model-profiles command is not available, and you do not need to select a model profile manually.

The supported FP16 pipelines are:

Serving Shape

Pipeline

Page elements

cuda-graph-fp16-page-elements

Table structure

cuda-graph-fp16-table-structure

The selected runtime path is logged at startup.

Precision#

FP16 is the supported precision for Object Detection NIM.

For standard deployments, leave NIM_ENGINE_PRECISION unset or set it to fp16.

docker run ... \
  -e NIM_ENGINE_PRECISION=fp16 \
  $IMG_NAME

Latency and Throughput#

Use NIM_PERFORMANCE_MODE to select latency-oriented or throughput-oriented defaults for unset variables.

Scenario

Recommended Settings

Lower latency, single-client workloads

NIM_PERFORMANCE_MODE=0, NIM_SERVER_MODE=latency, NIM_SERVER_MAX_WAIT_MS=0, NIM_ENGINE_COUNT=1, NIM_PIPELINE_MAX_BATCH_SIZE=1

Higher throughput, sustained concurrent workloads

NIM_PERFORMANCE_MODE=1, NIM_SERVER_MODE=throughput, NIM_SERVER_MAX_WAIT_MS=10, NIM_ENGINE_COUNT=2, NIM_PIPELINE_MAX_BATCH_SIZE=16

You can set any of the controlled variables explicitly to override the NIM_PERFORMANCE_MODE default.

Warmup and Runtime Artifact Cache#

The object detection runtime can compile and cache runtime artifacts under NIM_ENGINE_PRECOMPILE_CACHE_DIR. When unset, mount /opt/cache and let the container use the default cache location.

To precompile cuDNN plans and related runtime artifacts and then exit before the server starts, run the container with NIM_ENGINE_PRECOMPILE_ONLY=1.

docker run -it --rm \
  --runtime=nvidia \
  --gpus '"device=0"' \
  -e HF_TOKEN \
  -e NIM_ENGINE_MODEL_NAME="nvidia/nemotron-page-elements-v3" \
  -e NIM_ENGINE_MODEL_PATH="/model/page-elements" \
  -e NIM_ENGINE_WARMUP_MODE=full \
  -e NIM_ENGINE_PRECOMPILE_ONLY=1 \
  -v "$LOCAL_NIM_CACHE/cache:/opt/cache" \
  -v "$LOCAL_NIM_CACHE/weights:/model" \
  $IMG_NAME

NIM_ENGINE_WARMUP_MODE, NIM_ENGINE_WARMUP_SHAPES, NIM_ENGINE_CUDA_GRAPH_CAPTURE, and NIM_ENGINE_CUDA_GRAPH_CAPTURE_SHAPES provide advanced control over warmup and CUDA graph capture.

Workload

Recommended Settings

Variable batch sizes in production

NIM_ENGINE_WARMUP_MODE=full, NIM_ENGINE_CUDA_GRAPH_CAPTURE=full

Fixed batch size known ahead of time

NIM_ENGINE_WARMUP_MODE=partial, NIM_ENGINE_CUDA_GRAPH_CAPTURE_SHAPES=<batch-size>

Development, CI, or ephemeral startup

NIM_ENGINE_WARMUP_MODE=none

NIM_ENGINE_WARMUP_MODE=none starts faster, but the first requests for each batch size can be slower while cuDNN plans are selected inline. Use NIM_ENGINE_WARMUP_MODE=full for stable performance benchmarks and long-running production deployments with varied batch sizes.

NIM_PIPELINE_MAX_BATCH_SIZE is the GPU memory lever. Changing warmup or graph capture scope does not reduce the preallocated tensor buffer size.

Override Profile Selection#

Starting in version 2.0.0, the user-facing model profile workflow is not available. You cannot override the runtime pipeline selection by using NIM_MODEL_PROFILE.