API Reference for NVIDIA NIM for Object Detection

This documentation contains the API reference for NVIDIA NIM for Object Detection.

OpenAPI Specification

You can download the complete API spec.

Endpoints

The following endpoints are part of the OpenAPI contract.

Endpoint	Method	Description
/v1/page-elements	POST	Run page-elements object detection inference.
/v1/table-structure	POST	Run table-structure object detection inference.
/v1/models	GET	List the models loaded by the server.
/v1/health/live	GET	Check liveness.
/v1/health/ready	GET	Check readiness.
/v1/metrics	GET	Return Prometheus metrics.
/v1/metadata	GET	Return NIM metadata.
/v1/manifest	GET	Return model manifest metadata.
/v1/license	GET	Return license metadata and content.
/v1/version	GET	Return NIM release and API version.

The /v1/page-elements and /v1/table-structure endpoints are always mounted. If a deployment does not load the model for the requested endpoint, the server returns a 404 response that identifies the unloaded model and lists the available models.

Runtime Documentation Routes

The server also exposes the following convenience routes. These routes are not part of the OpenAPI contract and should not be treated as endpoints for generated inference clients.

Endpoint	Method	Description
/v1/openapi.json	GET	Return the runtime OpenAPI specification.
/openapi.json	GET	Return the runtime OpenAPI specification through the compatibility alias.
/docs	GET	Open the bundled Swagger UI.

Inference Request

The inference endpoints accept a JSON payload with an input array. Each item must be a JPEG or PNG image encoded as a data URL.

{
  "input": [
    {
      "type": "image_url",
      "url": "data:image/jpeg;base64,<BASE64_ENCODED_IMAGE>"
    }
  ]
}

The nested OpenAI-style image URL form is also supported.

{
  "input": [
    {
      "type": "image_url",
      "image_url": {
        "url": "data:image/png;base64,<BASE64_ENCODED_IMAGE>"
      }
    }
  ]
}

The request body does not include a model field. Choose the task by calling POST /v1/page-elements or POST /v1/table-structure.

The request supports the following optional fields.

Field	Type	Default	Description
confidence_threshold	number	0.01	Minimum confidence score for returned detections. The value must be between 0.0 and 1.0, inclusive.
nms_threshold	number	Model-specific	Non-maximum suppression threshold. When omitted, the runtime uses 0.5 for page elements and 0.25 for table structure. The value must be between 0.0 and 1.0, inclusive.

The API accepts data:image/jpeg;base64,... and data:image/png;base64,... inputs. External image URLs, plain base64 strings without a data URL prefix, GIF images, and inputs whose MIME type does not match the image bytes are rejected.

Note

The examples on this page include both JPEG and PNG input. For performance-sensitive workloads, use JPEG input to enable GPU-accelerated batched image decoding. For details, see Optimize Image Decoding.

Page Elements Example

API_ENDPOINT="http://localhost:8000"
IMAGE_SOURCE="https://assets.ngc.nvidia.com/products/api-catalog/nemo-retriever/object-detection/page-elements-example-1.jpg"
# IMAGE_SOURCE="path/to/your/image.jpg"  # Uncomment to use a local file instead.

if [[ "$IMAGE_SOURCE" == http* ]]; then
  BASE64_IMAGE=$(curl -sS "$IMAGE_SOURCE" | base64 -w 0)
else
  BASE64_IMAGE=$(base64 -w 0 "$IMAGE_SOURCE")
fi

MIME_TYPE="image/jpeg"
if [[ "$IMAGE_SOURCE" == *.png ]]; then
  MIME_TYPE="image/png"
fi

JSON_PAYLOAD='{
  "input": [{
    "type": "image_url",
    "url": "data:'${MIME_TYPE}';base64,'${BASE64_IMAGE}'"
  }]
}'

echo "${JSON_PAYLOAD}" | \
  curl -X POST "${API_ENDPOINT}/v1/page-elements" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d @-

The following image is used as the input in the previous example.

The response contains normalized bounding boxes and confidence scores for each detected label. Labels with no detections are omitted.

{
  "model": "nvidia/nemotron-page-elements-v3",
  "data": [
    {
      "index": 0,
      "bounding_boxes": {
        "table": [
          {
            "x_min": 0.36,
            "y_min": 0.2616,
            "x_max": 0.4907,
            "y_max": 0.3881,
            "confidence": 0.6416
          }
        ],
        "chart": [
          {
            "x_min": 0.2133,
            "y_min": 0.548,
            "x_max": 0.7816,
            "y_max": 0.8542,
            "confidence": 0.8397
          }
        ],
        "title": [
          {
            "x_min": 0.2384,
            "y_min": 0.1365,
            "x_max": 0.7192,
            "y_max": 0.1926,
            "confidence": 0.5737
          }
        ]
      }
    }
  ],
  "usage": {
    "images_size_mb": 0.10183906555175781
  }
}

The following image shows the input image with bounding boxes overlaid to visualize detected page elements.

Table Structure Example

API_ENDPOINT="http://localhost:8000"
IMAGE_SOURCE="https://assets.ngc.nvidia.com/products/api-catalog/nemo-retriever/object-detection/table-structure-example-1.png"
# IMAGE_SOURCE="path/to/your/image.png"  # Uncomment to use a local file instead.

if [[ "$IMAGE_SOURCE" == http* ]]; then
  BASE64_IMAGE=$(curl -sS "$IMAGE_SOURCE" | base64 -w 0)
else
  BASE64_IMAGE=$(base64 -w 0 "$IMAGE_SOURCE")
fi

JSON_PAYLOAD='{
  "input": [{
    "type": "image_url",
    "url": "data:image/png;base64,'${BASE64_IMAGE}'"
  }]
}'

echo "${JSON_PAYLOAD}" | \
  curl -X POST "${API_ENDPOINT}/v1/table-structure" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d @-

The following image is used as the input in the previous example.

The table-structure model returns bounding boxes for cell, row, and column detections.

Python Example

The following Python code demonstrates how to run inference and visualize the results.

Note

This example requires the requests and Pillow libraries. You can install them by using pip. For example: pip install requests Pillow

import base64
import io
import json
import os

import requests
from PIL import Image, ImageDraw


EXTENSION_TO_MIME = {
    ".jpg": "image/jpeg",
    ".jpeg": "image/jpeg",
    ".png": "image/png",
}


def encode_image(image_source):
    if image_source.startswith(("http://", "https://")):
        response = requests.get(image_source)
        response.raise_for_status()
        image_bytes = response.content
        _, ext = os.path.splitext(image_source.split("?")[0])
    else:
        with open(image_source, "rb") as f:
            image_bytes = f.read()
        _, ext = os.path.splitext(image_source)

    try:
        mime_type = EXTENSION_TO_MIME[ext.lower()]
    except KeyError:
        raise ValueError("Input image must be a JPEG or PNG image")

    encoded = base64.b64encode(image_bytes).decode("utf-8")
    return f"data:{mime_type};base64,{encoded}"


def detect_objects(image_data_url, api_endpoint, endpoint_path="/v1/page-elements"):
    payload = {
        "input": [
            {
                "type": "image_url",
                "url": image_data_url,
            }
        ]
    }
    response = requests.post(f"{api_endpoint}{endpoint_path}", json=payload)
    response.raise_for_status()
    return response.json()


def visualize_detections(image_data_url, result, output_path):
    b64_data = image_data_url.split(",", 1)[1]
    image_bytes = base64.b64decode(b64_data)
    image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
    draw = ImageDraw.Draw(image)
    width, height = image.size

    colors = {
        "table": "red",
        "chart": "green",
        "title": "blue",
        "infographic": "purple",
        "paragraph": "orange",
        "header_footer": "cyan",
        "cell": "red",
        "row": "green",
        "column": "blue",
    }

    for detection in result["data"]:
        for label, boxes in detection["bounding_boxes"].items():
            color = colors.get(label, "yellow")
            for box in boxes:
                x1 = int(box["x_min"] * width)
                y1 = int(box["y_min"] * height)
                x2 = int(box["x_max"] * width)
                y2 = int(box["y_max"] * height)
                draw.rectangle([x1, y1, x2, y2], outline=color, width=3)
                draw.text((x1, max(0, y1 - 15)), f"{label}: {box['confidence']:.2f}", fill=color)

    image.save(output_path)


if __name__ == "__main__":
    image_source = "https://assets.ngc.nvidia.com/products/api-catalog/nemo-retriever/object-detection/page-elements-example-1.jpg"
    api_endpoint = "http://localhost:8000"
    image_data_url = encode_image(image_source)
    result = detect_objects(image_data_url, api_endpoint, "/v1/page-elements")
    print(json.dumps(result, indent=2))
    visualize_detections(image_data_url, result, "detected_objects.jpg")

Error Handling

When you use the NVIDIA NIM for Object Detection NIM APIs, you might encounter the following errors.

Status Code	Description	Resolution
400	Bad request.	Verify that the JSON request body matches the schema.
404	The requested model endpoint is not available for the current deployment, or the route does not exist.	Verify that the target model is loaded. Use /v1/models to list the available models.
413	Payload too large.	Reduce image size or batch size.
415	Unsupported media type.	Set the Content-Type request header to application/json.
422	Unprocessable entity.	Verify that each image is a JPEG or PNG data URL and that the MIME type matches the image bytes.
503	Service unavailable.	Check readiness and wait for all loaded models to finish startup.
504	Gateway timeout.	Reduce request size or increase NIM_SERVER_REQUEST_TIMEOUT_S for cold-start or large-batch workloads.

Error responses use the following structure.

{
  "object": "error",
  "message": "request body did not match the expected schema",
  "type": "invalid_request_error",
  "detail": "..."
}

For example, if you call /v1/table-structure on a page-elements-only deployment, the server returns a 404 response similar to the following:

{
  "object": "error",
  "message": "model 'nvidia/nemotron-table-structure-v1' is not loaded; available models: nvidia/nemotron-page-elements-v3",
  "type": "not_found"
}

Health Check

curl "http://localhost:8000/v1/health/ready" \
  -H "Accept: application/json"

curl "http://localhost:8000/v1/health/live" \
  -H "Accept: application/json"

OpenAPI Reference