Is this page helpful?

Configure NVIDIA NIM for Object Detection#

NVIDIA NIM for Object Detection is packaged as a container. You can configure the NIM with Docker flags, mounted volumes, and environment variables.

Use this documentation to learn how to configure NVIDIA NIM for Object Detection.

Model Selection#

Use NIM_ENGINE_MODEL_NAME and NIM_ENGINE_MODEL_PATH to choose the object detection model that the container loads.

By default, the NIM container loads Page Elements with NIM_ENGINE_MODEL_NAME=nvidia/nemotron-page-elements-v3 and NIM_ENGINE_MODEL_PATH=/model/page-elements. To use Table Structure, set both variables to the Table Structure values in the following table when you launch the container.

Model	`NIM_ENGINE_MODEL_NAME`	`NIM_ENGINE_MODEL_PATH`	Inference Endpoint
Page elements	`nvidia/nemotron-page-elements-v3`	`/model/page-elements`	`/v1/page-elements`
Table structure	`nvidia/nemotron-table-structure-v1`	`/model/table-structure`	`/v1/table-structure`

Model selection occurs at container startup. To change the model, stop and recreate the container as described in Switch Between Page Elements and Table Structure.

Performance Mode#

NIM_PERFORMANCE_MODE selects latency-oriented or throughput-oriented defaults for variables that are not explicitly set.

Variable	Latency Default	Throughput Default
`NIM_SERVER_MAX_WAIT_MS`	`0`	`10`
`NIM_ENGINE_COUNT`	`1`	`2`
`NIM_PIPELINE_MAX_BATCH_SIZE`	`1`	`16`
`NIM_SERVER_MODE`	`latency`	`throughput`

Set a variable explicitly when you want it to take precedence over NIM_PERFORMANCE_MODE.

Warmup, CUDA Graph Capture, and VRAM#

NIM_PIPELINE_MAX_BATCH_SIZE sets the maximum number of images per engine forward pass and controls the size of the preallocated GPU tensor buffers. Supported values are 1 through 255. Reduce this value first when GPU memory is the limiting resource.

Warmup and CUDA graph settings control startup work and runtime dispatch coverage:

Workload	Recommended Settings
Variable batch sizes in production	`NIM_ENGINE_WARMUP_MODE=full`, `NIM_ENGINE_CUDA_GRAPH_CAPTURE=full`
Fixed batch size known ahead of time	`NIM_ENGINE_WARMUP_MODE=partial`, `NIM_ENGINE_CUDA_GRAPH_CAPTURE_SHAPES=<batch-size>`
Fast startup for development or CI	`NIM_ENGINE_WARMUP_MODE=none`

The default warmup settings are suitable for most deployments. Changing NIM_ENGINE_WARMUP_MODE or NIM_ENGINE_CUDA_GRAPH_CAPTURE does not reduce GPU memory usage because tensor buffers are still allocated for NIM_PIPELINE_MAX_BATCH_SIZE.

To warm or graph specific batch sizes, use comma-separated shape lists:

NIM_ENGINE_WARMUP_SHAPES=1,4,16
NIM_ENGINE_CUDA_GRAPH_CAPTURE_SHAPES=1,4,16

GPU Selection#

The NIM container is GPU-accelerated and uses NVIDIA Container Toolkit for access to GPUs on the host.

The documented Docker examples expose one GPU to the container with --gpus '"device=0"'. To use a different supported GPU, replace 0 with the device ID from nvidia-smi -L.

Refer to GPU Enumeration in the NVIDIA Container Toolkit documentation for more information.

Shared Memory Flag#

Tokenization uses capabilities that scale with the number of CPU cores available. You may need to increase the available shared memory given to the microservice container.

Example providing 1g of shared memory:

docker run ... --shm-size=1g ...

PID Limit#

In certain deployment or container runtime environments, default process and thread limits (PID limits) can interfere with NIM startup. These set limits are set by Docker, Podman, Kubernetes, or the operating system.

If the PID limit is too low, you might see symptoms such as:

NIM starts up partially, but fails to reach ready state, and then stalls.
NIM starts up partially, but fails to reach ready state, and then crashes.
NIM serves a small number of requests, and then fails.

To verify that PID limits are impacting the NIM container, you can remove or adjust the PID limit at the container, node, and operating system level. Removing the PID limit and then checking for success is a useful diagnostic step.

To increase the PID limit in a docker run command, set --pids-limit=-1. For details, see docker container run.
To increase the PID limit in a podman run command, set --pids-limit=-1. For details, see Podman pids-limit.
To increase the PID limit in Kubernetes, set the PodPidsLimit on the kubelet on each node. For details, see your Kubernetes distribution specific documentation.
To increase the PID limit at the operating system level, see your OS-specific documentation.

Volumes#

Mount model weights and runtime artifacts separately:

Mount model weights at /model.
Mount runtime artifacts, such as precompiled kernels and cuDNN plans, at /opt/cache.

docker run ... \
  -v "$LOCAL_NIM_CACHE/weights:/model" \
  -v "$LOCAL_NIM_CACHE/cache:/opt/cache" \
  $IMG_NAME

For page-elements inference, set NIM_ENGINE_MODEL_PATH=/model/page-elements. For table-structure inference, set NIM_ENGINE_MODEL_PATH=/model/table-structure.