Performance for NVIDIA NIM for Object Detection

Optimize Image Decoding

For the highest throughput, encode input images as JPEG. JPEG input enables GPU-accelerated batched image decoding, which is substantially more performant than processing PNG input, especially for requests that contain multiple images. PNG remains supported through the CPU image decode path, but use JPEG for performance-sensitive production workloads and representative throughput benchmarks.

Ensure that the media type in each data URL matches the encoded image bytes. Use data:image/jpeg;base64,... for JPEG input and data:image/png;base64,... for PNG input.

Benchmark Performance

To benchmark the performance of NVIDIA NIM for Object Detection, use a benchmark client that can send object detection requests to the endpoint you want to measure.

The runtime exposes model-specific inference endpoints:

• Page elements: POST /v1/page-elements

• Table structure: POST /v1/table-structure

The request body does not include a model field. Configure the benchmark client or wrapper to target the endpoint path for the loaded model.

Dataset

Create a dataset of representative image examples. The dataset should be formatted as a JSONL file where each line contains an image path, as shown in the following example.

Example: images.jsonl

{"image": "assets/image_01.jpg"}
{"image": "assets/image_02.jpg"}
{"image": "assets/image_n.jpg"}

Run GenAI-Perf from the directory that contains images.jsonl and the assets directory. The image_retrieval endpoint type reads each local image path and encodes the image as a base64 data URL in the object detection request.

To benchmark the page-elements endpoint, run the following command against a deployment that loaded nvidia/nemotron-page-elements-v3.

genai-perf profile \
  --model nvidia/nemotron-page-elements-v3 \
  --service-kind openai \
  --endpoint-type image_retrieval \
  --endpoint v1/page-elements \
  --batch-size-image 1 \
  --input-file images.jsonl \
  --concurrency 1 \
  --url http://localhost:8000

To benchmark the table-structure endpoint, run the following command against a deployment that loaded nvidia/nemotron-table-structure-v1.

genai-perf profile \
  --model nvidia/nemotron-table-structure-v1 \
  --service-kind openai \
  --endpoint-type image_retrieval \
  --endpoint v1/table-structure \
  --batch-size-image 1 \
  --input-file images.jsonl \
  --concurrency 1 \
  --url http://localhost:8000

Set --batch-size-image and --concurrency to the workload shape that you want to measure. The dataset must contain at least as many image entries as the configured image batch size.

Runtime Settings

For latency measurements, use latency-oriented server settings.

NIM_PERFORMANCE_MODE=0
NIM_SERVER_MODE=latency
NIM_SERVER_MAX_WAIT_MS=0
NIM_ENGINE_COUNT=1
NIM_PIPELINE_MAX_BATCH_SIZE=1

For throughput measurements, use throughput-oriented server settings.

NIM_PERFORMANCE_MODE=1
NIM_SERVER_MODE=throughput
NIM_SERVER_MAX_WAIT_MS=10
NIM_ENGINE_COUNT=2
NIM_PIPELINE_MAX_BATCH_SIZE=16

Use the same GPU, model, endpoint, batch size, concurrency, and server environment variables for each baseline and result run. For stable benchmark comparisons, use NIM_ENGINE_WARMUP_MODE=full; do not compare a run that uses NIM_ENGINE_WARMUP_MODE=none with a fully warmed run.

Example Benchmark Shapes

Use the following benchmark shapes as starting points.

Scenario	Endpoint	Image Batch Size	Concurrency
Page-elements latency	/v1/page-elements	1	1
Page-elements throughput	/v1/page-elements	16	8
Table-structure latency	/v1/table-structure	1	1
Table-structure throughput	/v1/table-structure	16	8

For the full set of command line options for genai-perf, refer to the GenAI-Perf documentation.