Deploy NVIDIA NIM for Nemotron Parse on Kubernetes#

You can deploy NVIDIA NIM for Nemotron Parse on Kubernetes by using Helm.

The Helm chart simplifies NIM deployment on Kubernetes. The chart supports deployment with a variety of possible cluster, GPU, and storage configurations.

NIM microservices run on hosts with NVIDIA GPUs. The GPU model and count depend on the model. To use Helm, you must have a Kubernetes cluster with GPU nodes.

To simplify cluster administration for GPU driver installation, consider installing NVIDIA GPU Operator to manage the driver installation and lifecycle.

Benefits of Helm Chart Deployment#

The benefits of Helm Charts are as follows:

Help manage the deployment and lifecycle of the microservice.
Use a centralized distribution mechanism for charts. The charts are hosted on NVIDIA NGC.
Enable you to customize the deployment when you specify a custom values file.

Setting Up the Environment#

If you haven’t set up your NGC API key and do not know exactly which NIM you want to download and deploy, see the information in the User Guide.

This Helm chart requires an image pull secret with your NGC API key, for pulling the container image from NGC. Whether you also need a model-download secret depends on the model download provider. For details, refer to Creating Secrets.

These instructions assume that you have your NGC_API_KEY exported in the environment.

export NGC_API_KEY="<YOUR NGC API KEY>"

Fetching the Helm Chart#

You can fetch the helm chart from NGC by executing the following command:

helm pull https://helm.ngc.nvidia.com/nim/nvidia/charts/nvidia-nim-nemotron-parse-2.4.0.tgz --username='$oauthtoken' --password=$NGC_API_KEY

Namespace#

You can choose to deploy to whichever namespace is appropriate, but for documentation purposes we will deploy to a namespace named nemotron-parse-nim.

kubectl create namespace nemotron-parse-nim

Creating Secrets#

Use the following procedure to create the secrets for this chart.

Add an image pull secret for downloading the container image from NVIDIA NGC. This secret is required for every deployment, including anonymous Hugging Face deployments.

kubectl create secret -n nemotron-parse-nim docker-registry ngc-secret \
 --docker-server=nvcr.io \
 --docker-username='$oauthtoken' \
 --docker-password="${NGC_API_KEY}"

Add a model-download secret if your model download provider needs one. Create one of the following secrets, or none:
- NGC: Add a generic secret that the container uses for downloading the model from NVIDIA NGC. The chart reads this secret by default.
```
kubectl create -n nemotron-parse-nim secret generic ngc-api --from-literal=NGC_API_KEY=${NGC_API_KEY}
```
- Hugging Face with a token: Add a generic secret with your Hugging Face token. To use it, refer to Deploying with Hugging Face.
```
kubectl create -n nemotron-parse-nim secret generic hf-api --from-literal=HF_TOKEN=${HF_TOKEN}
```
- Anonymous Hugging Face: Don’t create a model-download secret. For details, refer to Deploying with Hugging Face.

Configuration Considerations#

By default, the following deployment commands create a single deployment with one replica using the nemotron-parse container. For a description of all the Helm parameters, refer to Parameters. You can change this behavior with the following options:

image.repository – The container (nemotron-parse) to deploy.
image.tag – The version of that container (2.4).
Storage options, based on the environment and cluster in use. For details, refer to Storage.
resources – Use this option when a model requires more than the default of one GPU. For resource requirements, refer to the support matrix.
envVars – A map of environment variables to set in the container, if advanced configuration is needed. For example, --set envVars.NIM_PERFORMANCE_MODE=1. Don’t also set the env array: if env is set, the chart ignores envVars.

Storage#

This NIM downloads models at startup. Parse uses two cache locations:

/model/parse contains the downloaded model weights.
/opt/cache contains CUDA artifacts generated during warmup and inference.

For production use, persist both locations. The persistent deployment command in Deploying mounts one PVC at /model and /opt/cache. Persistent caches allow startup to reuse files.

Leave NIM_ENGINE_MODEL_PATH (default: /model/parse) and NIM_ENGINE_PRECOMPILE_CACHE_DIR (default: /opt/cache) unset to use these paths.

The following instructions use a local-nfs storage class provisioner. If your cluster already has a storage class provisioner, you can skip this step and specify your storage class in the deploy command.

helm repo add nfs-ganesha-server-and-external-provisioner https://kubernetes-sigs.github.io/nfs-ganesha-server-and-external-provisioner/
helm install nfs-server nfs-ganesha-server-and-external-provisioner/nfs-server-provisioner --set storageClass.name=local-nfs

Advanced Storage Configuration#

Storage is a particular concern when setting up NIMs. Models can be quite large, and you can fill a disk downloading things to emptyDirs or other locations around your pod image. We recommend that you mount persistent storage of some kind on your pod.

This chart supports two general categories:

Persistent Volume Claims (enabled by persistence.enabled)
hostPath (enabled by hostPath.enabled)

By default, persistence.enabled is set to false. With the cache mount settings in the deployment commands, both caches use an emptyDir volume unless persistent storage is enabled. To use persistent storage, set persistence.enabled=true in your Helm command.

When you set persistence.enabled=true, the chart creates a PersistentVolumeClaim that uses the cluster’s default storage class, or the class specified by persistence.storageClass.

If you want to store the models on an existing PersistentVolumeClaim, specify that value in persistence.existingClaim:

--set persistence.enabled=true \
--set persistence.existingClaim=my-existing-pvc

To use a local directory on the node instead of a PVC:

--set persistence.enabled=false \
--set hostPath.enabled=true \
--set hostPath.path=/path/to/local/model-store

Keep both cache mount settings from the deployment commands when using an existing PVC or a hostPath. The hostPath directory must be accessible and writable by the container user, UID 1000 and GID 1000.

You can also supply the cache settings in a values file. Save the following as parse-values.yaml:

persistence:
  enabled: true
nim:
  nimCache: /model
extraVolumeMounts:
  model-store:
    mountPath: /opt/cache

Pass the file to Helm and select the storage class for your cluster:

helm upgrade --install \
  --namespace nemotron-parse-nim \
  nemotron-parse \
  -f parse-values.yaml \
  --set persistence.storageClass="local-nfs" \
  nvidia-nim-nemotron-parse-2.4.0.tgz

For more information, refer to Parameters.

Deploying#

The commands in this section download the model from NGC by using the ngc-api secret. To download the model from Hugging Face instead, refer to Deploying with Hugging Face.

Use the following command to create a deployment with persistent storage for both model weights and CUDA artifacts.

helm upgrade --install \
  --namespace nemotron-parse-nim \
  nemotron-parse \
  --set persistence.enabled=true \
  --set persistence.storageClass="local-nfs" \
  --set nim.nimCache=/model \
  --set extraVolumeMounts.model-store.mountPath=/opt/cache \
  nvidia-nim-nemotron-parse-2.4.0.tgz

Use the following command to deploy without persistent storage. Both caches use an emptyDir volume: files survive container restarts within the same pod but are lost when the pod is deleted or replaced.

helm upgrade --install \
  --namespace nemotron-parse-nim \
  nemotron-parse \
  --set persistence.enabled=false \
  --set nim.nimCache=/model \
  --set extraVolumeMounts.model-store.mountPath=/opt/cache \
  nvidia-nim-nemotron-parse-2.4.0.tgz

You can also change the version of the model in use by adding the following after --namespace.

--set image.tag=2.4 \

After deploying check the pods to ensure that it is running, initial image pull and model download can take upwards of 15 minutes.

kubectl get pods -n nemotron-parse-nim

The pod should eventually end up in the Running state. You should see output similar to the following.

NAME                              READY   STATUS    RESTARTS   AGE
nemotron-parse-nvidia-nim-nemotron-parse-0   1/1     Running   0          8m44s

Check events for failures:

kubectl get events -n nemotron-parse-nim

Deploying with Hugging Face#

To download the model from Hugging Face, select the hf provider and set nim.ngcAPISecret to an empty string, so the chart doesn’t inject an NGC model-download key. Keep imagePullSecrets, because the container image still comes from NGC.

Set the provider in the env array, together with any other environment variables. If env is set, the chart ignores envVars.

Hugging Face with a Token#

Create the hf-api secret as described in Creating Secrets. Then save the following as parse-hf-values.yaml:

imagePullSecrets:
  - name: ngc-secret
nim:
  ngcAPISecret: ""
env:
  - name: NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER
    value: hf
  - name: HF_TOKEN
    valueFrom:
      secretKeyRef:
        name: hf-api
        key: HF_TOKEN

Anonymous Hugging Face#

An anonymous deployment doesn’t use a model-download credential. Save the following as parse-hf-values.yaml:

imagePullSecrets:
  - name: ngc-secret
nim:
  ngcAPISecret: ""
env:
  - name: NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER
    value: hf

Deploy with the Values File#

Use the following command to deploy with persistent storage and the values file.

helm upgrade --install \
  --namespace nemotron-parse-nim \
  nemotron-parse \
  -f parse-hf-values.yaml \
  --set persistence.enabled=true \
  --set persistence.storageClass="local-nfs" \
  --set nim.nimCache=/model \
  --set extraVolumeMounts.model-store.mountPath=/opt/cache \
  nvidia-nim-nemotron-parse-2.4.0.tgz

To check that the deployment is running, use the kubectl get pods command from Deploying.

Recommended Configuration for Minikube#

Minikube creates a hostPath-based PV and PVC when persistence is enabled. We recommend that you add the following to your helm commands.

--set persistence.enabled=true \
--set persistence.storageClass=standard

Running Inference#

In the previous example the API endpoint is exposed on port 8000 through the Kubernetes service of the default type with no ingress, since authentication is not handled by the NIM itself. Port 8000 is the default service.httpPort. If you set a different service.httpPort, use that value as the second port in the following kubectl port-forward command; the container still listens on port 8000. The following commands require that the nvidia/nemotron-3.5-parse model has been deployed.

Use the following command to port-forward the service to your local computer to test inference.

kubectl port-forward -n nemotron-parse-nim service/nemotron-parse-nvidia-nim-nemotron-parse 8000:8000

Send one rendered document page to the /v1/parse endpoint.

curl -X POST "http://localhost:8000/v1/parse" \
  -H "Accept: application/json" \
  -F "document=@page.png;type=image/png"

For more request examples, refer to API Usage Examples (OpenAPI) for NVIDIA NIM for Nemotron Parse.

Viewing Log Messages#

Use the following command to view the container log messages.

# Step 1: Find the actual pod name
kubectl get pods -n nemotron-parse-nim

# Step 2: Use the actual pod name from step 1
kubectl logs -f -n nemotron-parse-nim <pod-name-from-step-1>