Performance and Memory for NVIDIA NIM for Object Detection#

Optimize Image Decoding#

For the highest throughput, encode input images as JPEG. JPEG input enables GPU-accelerated batched image decoding, which is substantially more performant than processing PNG input, especially for requests that contain multiple images. PNG remains supported through the CPU image decode path, but use JPEG for performance-sensitive production workloads and representative throughput benchmarks.

Ensure that the media type in each data URL matches the encoded image bytes. Use data:image/jpeg;base64,... for JPEG input and data:image/png;base64,... for PNG input.

Benchmark Performance#

To benchmark the performance of NVIDIA NIM for Object Detection, use a benchmark client that can send object detection requests to the endpoint you want to measure.

The runtime exposes model-specific inference endpoints:

Page elements: POST /v1/page-elements
Table structure: POST /v1/table-structure

The request body does not include a model field. Configure the benchmark client or wrapper to target the endpoint path for the loaded model.

Dataset#

Create a dataset of representative image examples. The dataset should be formatted as a JSONL file where each line contains an image path, as shown in the following example.

Example: images.jsonl

{"image": "assets/image_01.jpg"}
{"image": "assets/image_02.jpg"}
{"image": "assets/image_n.jpg"}

Run GenAI-Perf from the directory that contains images.jsonl and the assets directory. The image_retrieval endpoint type reads each local image path and encodes the image as a base64 data URL in the object detection request.

To benchmark the page-elements endpoint, run the following command against a deployment that loaded nvidia/nemotron-page-elements-v3.

genai-perf profile \
  --model nvidia/nemotron-page-elements-v3 \
  --service-kind openai \
  --endpoint-type image_retrieval \
  --endpoint v1/page-elements \
  --batch-size-image 1 \
  --input-file images.jsonl \
  --concurrency 1 \
  --url http://localhost:8000

To benchmark the table-structure endpoint, run the following command against a deployment that loaded nvidia/nemotron-table-structure-v1.

genai-perf profile \
  --model nvidia/nemotron-table-structure-v1 \
  --service-kind openai \
  --endpoint-type image_retrieval \
  --endpoint v1/table-structure \
  --batch-size-image 1 \
  --input-file images.jsonl \
  --concurrency 1 \
  --url http://localhost:8000

Set --batch-size-image and --concurrency to the workload shape that you want to measure. The dataset must contain at least as many image entries as the configured image batch size.

Runtime Settings#

For latency measurements, use latency-oriented server settings.

NIM_PERFORMANCE_MODE=0
NIM_SERVER_MODE=latency
NIM_SERVER_MAX_WAIT_MS=0
NIM_ENGINE_COUNT=1
NIM_PIPELINE_MAX_BATCH_SIZE=1

For throughput measurements, use throughput-oriented server settings.

NIM_PERFORMANCE_MODE=1
NIM_SERVER_MODE=throughput
NIM_SERVER_MAX_WAIT_MS=10
NIM_ENGINE_COUNT=2
NIM_PIPELINE_MAX_BATCH_SIZE=16

Use the same GPU, model, endpoint, batch size, concurrency, and server environment variables for each baseline and result run. For stable benchmark comparisons, use NIM_ENGINE_WARMUP_MODE=full; do not compare a run that uses NIM_ENGINE_WARMUP_MODE=none with a fully warmed run.

Example Benchmark Shapes#

Use the following benchmark shapes as starting points.

Scenario	Endpoint	Image Batch Size	Concurrency
Page-elements latency	`/v1/page-elements`	`1`	`1`
Page-elements throughput	`/v1/page-elements`	`16`	`8`
Table-structure latency	`/v1/table-structure`	`1`	`1`
Table-structure throughput	`/v1/table-structure`	`16`	`8`

For the full set of command line options for genai-perf, refer to the GenAI-Perf documentation.

About the Measurements#

The following sections report the memory footprint for each model. The approximate GPU memory sizes are the startup measurements from the latency configuration after forward buffers are allocated. Runtime GPU memory use can increase with larger inputs and input-dependent working buffers.

Note

GB10 uses unified system memory. GPU memory usage is reported as N/A because the usual VRAM measurements are not accurate for this architecture. For more information, refer to Guidance for Reporting Memory Resources with Unified Memory Architecture.

For the GPU SKUs and precisions that the NIM supports, refer to Support Matrix for NVIDIA NIM for Object Detection.

Nemotron Page Elements v3#

Memory Footprint#

The following table contains the memory footprint data for Nemotron Page Elements v3.

GPU	Approximate GPU Memory Size (GiB)
NVIDIA A100 SXM4 80GB	1.05
NVIDIA A10G	0.88
NVIDIA B200	1.26
NVIDIA GB10 Grace Blackwell Superchip, such as NVIDIA DGX Spark systems	N/A
NVIDIA GB200	1.32
NVIDIA H100 80GB HBM3	1.15
NVIDIA H100 NVL	1.15
NVIDIA H200	1.15
NVIDIA L4	0.82
NVIDIA L40S	1.06
NVIDIA RTX PRO 6000 Blackwell Server Edition	1.19

Nemotron Table Structure v1#

Memory Footprint#

The following table contains the memory footprint data for Nemotron Table Structure v1.

GPU	Approximate GPU Memory Size (GiB)
NVIDIA A100 SXM4 80GB	1.05
NVIDIA A10G	0.88
NVIDIA B200	1.26
NVIDIA GB10 Grace Blackwell Superchip, such as NVIDIA DGX Spark systems	N/A
NVIDIA GB200	1.32
NVIDIA H100 80GB HBM3	1.15
NVIDIA H100 NVL	1.15
NVIDIA H200	1.15
NVIDIA L4	0.82
NVIDIA L40S	1.06
NVIDIA RTX PRO 6000 Blackwell Server Edition	1.19