Is this page helpful?

Get Started With NVIDIA NIM for Image OCR (NeMo Retriever OCR)#

This documentation helps you get started with NVIDIA NIM for Image OCR (NeMo Retriever OCR).

Prerequisites#

Before you can get started, you need the following:

Verify that you have supported hardware and software. For details, refer to the support matrix.
If you are running on an RTX AI PC or Workstation, install WSL2. For instructions, refer to NIM on WSL2 documentation.
Create an account on NVIDIA NGC and generate an API key to access the NIM container images and model assets on NGC. For instructions, refer to the NGC Authentication section that follows.

Note

Deploying on Kubernetes is not supported for WSL.

NGC Authentication#

Generate your API key#

To access the NIM container images and model assets on NGC, you must generate a personal API key. To create your key, go to https://org.ngc.nvidia.com/setup/api-keys.

When you create your key, for Services Included, select the following:

NGC Catalog
Private Registry (if you are an Early Access participant)

You can include more services if you are going to use this key for other purposes. For more information, refer to the NGC User Guide.

Export the API key#

To conveniently use your API key in the commands in the following sections, you can export your key as an environment variable named NGC_API_KEY. For example, run the following code in your terminal.

export NGC_API_KEY=<your API key value>

Run one of the following commands to make your key available when you start a new terminal session.

# If using bash
echo "export NGC_API_KEY=<value>" >> ~/.bashrc

# If using zsh
echo "export NGC_API_KEY=<value>" >> ~/.zshrc

Note

Other, more secure options include saving your API key value in a file (retrieve it by using cat $NGC_API_KEY_FILE), or saving your key in a password manager.

Docker Login to NGC#

Before you can pull the NIM container image from NGC, first authenticate to NGC by using the following command.

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Use $oauthtoken as the username and NGC_API_KEY as the password. The $oauthtoken username is a special name that indicates that you will authenticate with an API key and not a user name and password.

Accept the License Terms#

Some NIM models require that you accept the license terms on NGC before you can pull the container image and model assets. To accept the license terms, browse to the model or container page on the NGC Catalog, read and then click Accept Terms.

For example, accept terms for the following container:

nemotron-ocr-v2

Launch the NIM#

For Hugging Face model downloads, set both NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=hf and HF_TOKEN. The following example uses NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc and the NGC_API_KEY configured in the prerequisites to download model artifacts from NGC.

export NIM_MODEL_NAME=nvidia/nemotron-ocr-v2
export CONTAINER_NAME=$(basename "$NIM_MODEL_NAME")
export IMG_NAME="nvcr.io/nim/nvidia/nemotron-ocr-v2:2.0"
export LOCAL_NIM_CACHE=~/.cache/nim

mkdir -p "$LOCAL_NIM_CACHE/cache" "$LOCAL_NIM_CACHE/weights"

docker run -it --rm --name="$CONTAINER_NAME" \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=16GB \
  -e NGC_API_KEY \
  -e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc \
  -v "$LOCAL_NIM_CACHE/cache:/opt/cache" \
  -v "$LOCAL_NIM_CACHE/weights:/model" \
  -u $(id -u) \
  -p 8000:8000 \
  "$IMG_NAME"

Verify that the Service is Ready#

After you launch the NIM, it might take a few seconds for the service to be ready to accept requests. To verify that the service is ready, run the following code.

curl -X 'GET' 'http://localhost:8000/v1/health/ready'

If the service is ready, you should see a response similar to the following.

{"object":"health.response","message":"ready","ready":true}

Run Inference#

After the service is ready, use code similar to the following to run inference. For more information, refer to API Reference for NVIDIA NIM for Image OCR (NeMo Retriever OCR).

Note

This example uses a PNG sample image. For performance-sensitive workloads, use JPEG input to enable batched image decoding. For details, see Optimize Image Decoding.

API_ENDPOINT="http://localhost:8000"

# Create JSON payload with base64 encoded image
IMAGE_SOURCE="https://assets.ngc.nvidia.com/products/api-catalog/nemo-retriever/image-ocr/example-1.png"
# IMAGE_SOURCE="path/to/your/image.png"  # Uncomment to use a local file instead

# Encode the image to base64 (handles both URLs and local files)
if [[ $IMAGE_SOURCE == http* ]]; then
  # Handle URL
  BASE64_IMAGE=$(curl -s ${IMAGE_SOURCE} | base64 -w 0)
else
  # Handle local file
  BASE64_IMAGE=$(base64 -w 0 ${IMAGE_SOURCE})
fi

# Construct the full JSON payload
JSON_PAYLOAD='{
  "input": [{
    "type": "image_url",
    "url": "data:image/png;base64,'${BASE64_IMAGE}'"
  }],
  "merge_levels": ["word"]
}'

# Send POST request to inference endpoint
echo "${JSON_PAYLOAD}" | \
  curl -X POST "${API_ENDPOINT}/v1/ocr" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d @-

If the request succeeds, you should see a response similar to the following.

{
  "model": "nvidia/nemotron-ocr-v2",
  "data": [
    {
      "index": 0,
      "text_detections": [
        {
          "text_prediction": {
            "text": "Example text",
            "confidence": 0.99
          },
          "bounding_box": {
            "points": [
              {"x": 0.10, "y": 0.20},
              {"x": 0.60, "y": 0.20},
              {"x": 0.60, "y": 0.40},
              {"x": 0.10, "y": 0.40}
            ]
          }
        }
      ]
    }
  ],
  "usage": {
    "images_size_mb": 0.1
  }
}

Deploy on Multiple GPUs#

Multi-GPU deployment is not supported in release 2.0.0. NIM_ENGINE_DEVICES selects one CUDA-visible device ordinal for OCR engine placement.

Deploy Alongside Other NIMs on the Same GPU#

You can deploy Image OCR NIM (NeMo Retriever OCR) alongside another NIM (for example, llama3-8b-instruct) on the same GPU (for example, A100 80GB, A100 40GB, or H100 80GB). For more information about deployment, see Get Started with NIM LLM, NIM Operator, GPU Operator with MIG, and Time-Slicing GPUs in Kubernetes.

Use the docker run --gpus command-line argument to specify the same GPU as shown in the following code.

docker run --gpus '"device=1"' ... $IMG_NAME
docker run --gpus '"device=1"' ... <your-llm-image, such as nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.5>

Download NIM Models to Cache#

Use download-only mode to stage the model without CUDA initialization or server startup.

export IMG_NAME="nvcr.io/nim/nvidia/nemotron-ocr-v2:2.0"
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE/weights"

docker run -it --rm \
  -e NIM_ENGINE_MODEL_DOWNLOAD_ONLY=1 \
  -e NGC_API_KEY \
  -e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc \
  -v "$LOCAL_NIM_CACHE/weights:/model" \
  -u $(id -u) \
  "$IMG_NAME"

The command stores the OCR model under $LOCAL_NIM_CACHE/weights/ocr. For an air-gapped deployment, transfer the weights directory to the target host, mount it at /model, mount a writable runtime cache at /opt/cache, and do not pass provider credentials.

Stop the Container#

To stop the Docker container, run the following code.

docker stop $CONTAINER_NAME

To remove the Docker container, run the following code. If you included the --rm flag when you started the container, you don’t need this step.

docker rm $CONTAINER_NAME

Get Started With NVIDIA NIM for Image OCR (NeMo Retriever OCR)#

Prerequisites#

NGC Authentication#

Generate your API key#

Export the API key#

Docker Login to NGC#

Accept the License Terms#

Launch the NIM#

Verify that the Service is Ready#

Run Inference#

Deploy on Multiple GPUs#

Deploy Alongside Other NIMs on the Same GPU#

Download NIM Models to Cache#

Stop the Container#

Related Topics#