Is this page helpful?

Get Started With NVIDIA NIM for Nemotron Parse#

This documentation helps you get started with NVIDIA NIM for Nemotron Parse.

Prerequisites#

Before you can get started, you need the following:

Verify that you have supported hardware and software. For details, refer to the support matrix.
If you are running on an RTX AI PC or Workstation, install WSL2. For instructions, refer to NIM on WSL2 documentation.
Create an account on NVIDIA NGC and generate an API key to access the NIM container images and model assets on NGC. For instructions, refer to the NGC Authentication section that follows.

Note

Deploying on Kubernetes is not supported for WSL.

NGC Authentication#

Generate your API key#

To access the NIM container images and model assets on NGC, you must generate a personal API key. To create your key, go to https://org.ngc.nvidia.com/setup/api-keys.

When you create your key, for Services Included, select the following:

NGC Catalog
Private Registry (if you are an Early Access participant)

You can include more services if you are going to use this key for other purposes. For more information, refer to the NGC User Guide.

Export the API key#

To conveniently use your API key in the commands in the following sections, you can export your key as an environment variable named NGC_API_KEY. For example, run the following code in your terminal.

export NGC_API_KEY=<your API key value>

Run one of the following commands to make your key available when you start a new terminal session.

# If using bash
echo "export NGC_API_KEY=<value>" >> ~/.bashrc

# If using zsh
echo "export NGC_API_KEY=<value>" >> ~/.zshrc

Note

Other, more secure options include saving your API key value in a file (retrieve it by using cat $NGC_API_KEY_FILE), or saving your key in a password manager.

Docker Login to NGC#

Before you can pull the NIM container image from NGC, first authenticate to NGC by using the following command.

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Use $oauthtoken as the username and NGC_API_KEY as the password. The $oauthtoken username is a special name that indicates that you will authenticate with an API key and not a user name and password.

Accept the License Terms#

Some NIM models require that you accept the license terms on NGC before you can pull the container image and model assets. To accept the license terms, browse to the model or container page on the NGC Catalog, read and then click Accept Terms.

For example, accept terms for the following container:

nemotron-parse

Model Download Providers#

Nemotron Parse NIM downloads the model weights from one of the following providers:

NGC: The Nemotron 3.5 Parse model in the NGC Catalog. Use the same NGC API key that you use for the container image.
Hugging Face: The Nemotron 3.5 Parse Hugging Face repository. You can download anonymously, without a token, or use a Hugging Face token with read access to the repository.

The NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER environment variable selects the provider. With the default value, auto, the NIM uses Hugging Face when HF_TOKEN is set, and otherwise uses NGC when NGC_API_KEY is set. If neither is set, the NIM tries Hugging Face without a token. To always use one provider, set NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER to ngc or hf.

To use Hugging Face with a token, run the following code to export your token before you launch the container:

export HF_TOKEN="<your-hugging-face-token>"

To use NGC, pass the NGC_API_KEY that you exported in NGC Authentication.

Launch the NIM#

The following commands launch the nemotron-parse NIM container, which serves the nvidia/nemotron-3.5-parse model.

The NIM downloads and verifies the supported model checkpoint if it is not already cached. To download from NGC, pass NGC_API_KEY. To download from Hugging Face, pass HF_TOKEN, or select the hf provider and pass no model-download credential for an anonymous download. You still need to log in to nvcr.io to pull the container image. For details, refer to Model Download Providers.

# Choose a container name for bookkeeping
export CONTAINER_NAME=nemotron-parse

# Choose a NIM Image from NGC
export IMG_NAME="nvcr.io/nim/nvidia/nemotron-parse:2.4"

# Choose a path on your system to cache model weights and compiled artifacts
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE/weights" "$LOCAL_NIM_CACHE/precompile-cache"

Mount the model weights cache at /model. The NIM downloads the weights to /model/parse, so later runs reuse them instead of downloading them again. Leave NIM_ENGINE_MODEL_PATH unset to use this default path.

Hugging Face Provider with a Token#

docker run -it --rm --name=$CONTAINER_NAME \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=16GB \
  -e HF_TOKEN \
  -v "$LOCAL_NIM_CACHE/weights:/model" \
  -v "$LOCAL_NIM_CACHE/precompile-cache:/opt/cache" \
  -u $(id -u) \
  -p 8000:8000 \
  $IMG_NAME

Anonymous Hugging Face Provider#

To download from Hugging Face without a token, select the hf provider and don’t pass HF_TOKEN or NGC_API_KEY.

docker run -it --rm --name=$CONTAINER_NAME \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=16GB \
  -e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=hf \
  -v "$LOCAL_NIM_CACHE/weights:/model" \
  -v "$LOCAL_NIM_CACHE/precompile-cache:/opt/cache" \
  -u $(id -u) \
  -p 8000:8000 \
  $IMG_NAME

NGC Provider#

docker run -it --rm --name=$CONTAINER_NAME \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=16GB \
  -e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE/weights:/model" \
  -v "$LOCAL_NIM_CACHE/precompile-cache:/opt/cache" \
  -u $(id -u) \
  -p 8000:8000 \
  $IMG_NAME

The following table describes the flags in these commands.

Flags	Description
`-it`	`--interactive` + `--tty` (see Docker docs).
`--rm`	Delete the container after it stops.
`--name=$CONTAINER_NAME`	Give a name to the NIM container for bookkeeping.
`--runtime=nvidia`	Ensure NVIDIA drivers are accessible in the container.
`--gpus '"device=0"'`	Expose one GPU inside the container. Nemotron Parse NIM runs on a single GPU.
`--shm-size=16GB`	Allocate shared memory for the container.
`-e HF_TOKEN`	Hugging Face access token. Used to download the model weights from Hugging Face.
`-e NGC_API_KEY`	NGC API key. Used to download the model weights from NGC.
`-e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=hf`	Always download the model weights from Hugging Face. Without `HF_TOKEN`, the download is anonymous.
`-e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc`	Always download the model weights from NGC, even when `HF_TOKEN` is set.
`-v "$LOCAL_NIM_CACHE/weights:/model"`	Mount a model weights cache directory, so later runs reuse the downloaded weights.
`-u $(id -u)`	Use the same user as your system user inside the container to avoid permission mismatches on the cache directory.
`-p 8000:8000`	Publish the NIM HTTP port.
`$IMG_NAME`	NIM container image from NGC.

Verify that the Service is Ready#

After you launch the NIM, model download, loading, and GPU warmup can take time before the service is ready to accept requests. To verify that the service is ready, run the following code.

curl -X 'GET' 'http://localhost:8000/v1/health/ready'

If the service is ready, you should see a response similar to the following.

{"object":"health.response","message":"ready","ready":true}

Run Inference#

After the service is ready, send one rendered document page as an image to the /v1/parse endpoint. For more information, refer to API Usage Examples (OpenAPI) for NVIDIA NIM for Nemotron Parse.

curl -X POST "http://localhost:8000/v1/parse" \
  -H "Accept: application/json" \
  -F "document=@page.png;type=image/png"

If the request succeeds, you should see a response similar to the following. The content field contains extracted text with Markdown formatting and coordinate/class control tokens. These tokens are expected in the response.

{
  "content": "<x_0.0254><y_0.0297>Nemotron Parse documentation smoke test<x_0.2754><y_0.043><class_Text>\n\n<x_0.0254><y_0.0437>A document page with readable text<x_0.2354><y_0.0563><class_Text>",
  "finish_reason": "stop",
  "usage": {
    "output_tokens": 26
  }
}

Deploy on Multiple GPUs#

Nemotron Parse NIM runs on a single GPU, and deploying one NIM across multiple GPUs isn’t supported. To handle more requests, run one NIM container on each GPU and balance requests across them. To tune concurrency within a container, configure decoder capacity as described in Optimization for NVIDIA NIM for Nemotron Parse.

Deploy Alongside Other NIMs on the Same GPU#

You can deploy Nemotron Parse alongside another NIM (for example, llama3-8b-instruct) on the same GPU (for example, A100 80GB or H100 80GB). For more information about deployment, see Get Started with NIM LLM, NIM Operator, GPU Operator with MIG, and Time-Slicing GPUs in Kubernetes.

Use the docker run --gpus command-line argument to specify the same GPU as shown in the following code.

docker run --gpus '"device=1"' ... $IMG_NAME
docker run --gpus '"device=1"' ... <your-llm-image, such as nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.5>

Download the Model Weights in Advance#

To download the model weights without starting the server, for example to prepare an air-gapped host, run the container with NIM_ENGINE_MODEL_DOWNLOAD_ONLY=1. The NIM downloads and checks the weights, and then exits. This run doesn’t need a GPU.

docker run -it --rm --name=$CONTAINER_NAME \
  -e NIM_ENGINE_MODEL_DOWNLOAD_ONLY=1 \
  -e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE/weights:/model" \
  -u $(id -u) \
  $IMG_NAME

To download from Hugging Face instead, replace the NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER and NGC_API_KEY lines with -e HF_TOKEN. For an anonymous Hugging Face download, replace them with -e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=hf.

After the command exits successfully, the verified model files are in $LOCAL_NIM_CACHE/weights/parse. Stage the weights with the exact container image that will serve the model, then copy the parent weights directory to the offline host.

The checkpoint for this release is pinned to revision dca5b2df89d00a671509204501d8f1f379d5e94f and requires the following six files:

config.json
model.safetensors
tokenizer.json
LICENSE
THIRD_PARTY_NOTICES.md
TOKENIZER_ATTRIBUTION.md

If any of these files is missing, the NIM tries to download it, which fails on an offline host. Startup checks file sizes and SHA-256 hashes. A different checkpoint, a corrupt nonempty file, or an incomplete cache can prevent startup. To replace an incompatible cache, stage the supported checkpoint in a clean directory and update the mount. The NIM does not automatically overwrite an incompatible nonempty file.

On the air-gapped host, load the same container image, mount the staged parent weights directory at /model, and start the NIM without NGC_API_KEY or HF_TOKEN. A complete verified cache does not require a model download or model-download credentials.

Stop the Container#

To stop the Docker container, run the following code.

docker stop $CONTAINER_NAME

To remove the Docker container, run the following code. If you included the --rm flag when you started the container, you don’t need this step.

docker rm $CONTAINER_NAME