Deploy NVIDIA NeMo Retriever Embedding NIM on Kubernetes#

You can deploy NVIDIA NeMo Retriever Embedding NIM on Kubernetes by using Helm.

The Helm chart simplifies NIM deployment on Kubernetes. The chart supports deployment with a variety of possible cluster, GPU, and storage configurations.

NIM microservices run on hosts with NVIDIA GPUs. The GPU model and count depend on the model. To use Helm, you must have a Kubernetes cluster with GPU nodes.

To simplify cluster administration for GPU driver installation, consider installing NVIDIA GPU Operator to manage the driver installation and lifecycle.

Benefits of Helm Chart Deployment#

The benefits of Helm Charts are as follows:

Help manage the deployment and lifecycle of the microservice.
Use a centralized distribution mechanism for charts. The charts are hosted on NVIDIA NGC.
Enable you to customize the deployment when you specify a custom values file.

Setting Up the Environment#

If you haven’t set up your NGC API key and do not know exactly which NIM you want to download and deploy, see the information in the User Guide.

This helm chart requires that you have a secret with your NGC API key configured for downloading private images, and one with your NGC API key (below named ngc-api). These will likely have the same key in it, but they will have different formats (dockerconfig.json vs opaque). See Creating Secrets below.

These instructions will assume that you have your NGC_API_KEY exported in the environment.

export NGC_API_KEY="<YOUR NGC API KEY>"

Fetching the Helm Chart#

You can fetch the helm chart from NGC by executing the following command:

helm pull https://helm.ngc.nvidia.com/nim/nvidia/charts/nvidia-nim-nemotron-embed-2.4.0.tgz --username='$oauthtoken' --password=$NGC_API_KEY

Namespace#

You can choose to deploy to whichever namespace is appropriate, but for documentation purposes we will deploy to a namespace named embedding-nim.

kubectl create namespace embedding-nim

Creating Secrets#

Use the following procedure to create the secrets for this chart.

Add an image pull secret for downloading the container image from NVIDIA NGC.

kubectl create secret -n embedding-nim docker-registry ngc-secret \
 --docker-server=nvcr.io \
 --docker-username='$oauthtoken' \
 --docker-password="${NGC_API_KEY}"

Add a generic secret that the container uses for downloading models from NVIDIA NGC.

kubectl create -n embedding-nim secret generic ngc-api --from-literal=NGC_API_KEY=${NGC_API_KEY}

Configuration Considerations#

By default, the following deployment commands create a single deployment with one replica using the nemotron-embed container. For a description of all the Helm parameters, refer to Parameters. You can change this behavior with the following options:

image.repository – The container (nemotron-embed) to deploy.
image.tag – The version of that container (2.4.0).
Storage options, based on the environment and cluster in use.
resources – Use this option when a model requires more than the default of one GPU. For resource requirements, refer to the support matrix.
envVars – A map of environment variables to set in the container, if advanced configuration is needed. For example, --set envVars.NIM_PERFORMANCE_MODE=1. Don’t also set the env array: if env is set, the chart ignores envVars.

Storage#

Nemotron Embed 2.4 stores model weights in /model/embed and compiler and runtime caches in /opt/cache. By default, the chart mounts model-store at /opt/nim/.cache, which doesn’t cover these paths. Setting persistence.enabled=true alone doesn’t correct the mount paths.

Use both settings in every deployment:

--set nim.nimCache=/model \
--set extraVolumeMounts.model-store.mountPath=/opt/cache

These settings mount the same volume at /model and /opt/cache. With persistent storage enabled, the volume retains model weights and compiler and runtime caches when a Pod is replaced. When both persistence.enabled and hostPath.enabled are false, the chart uses an emptyDir volume, and replacing the Pod loses its contents.

The following deployment example uses a local-nfs storage class provisioner. If your cluster already has a storage class, skip this step and specify your storage class in the deploy command. To install the provisioner, run the following commands:

helm repo add nfs-ganesha-server-and-external-provisioner https://kubernetes-sigs.github.io/nfs-ganesha-server-and-external-provisioner/
helm install nfs-server nfs-ganesha-server-and-external-provisioner/nfs-server-provisioner --set storageClass.name=local-nfs

Advanced Storage Configuration#

To use an existing PersistentVolumeClaim, add the following options to the Helm command:

--set persistence.enabled=true \
--set persistence.existingClaim=my-existing-pvc \
--set nim.nimCache=/model \
--set extraVolumeMounts.model-store.mountPath=/opt/cache

For node-local storage, use the following options instead:

--set persistence.enabled=false \
--set hostPath.enabled=true \
--set hostPath.path=/path/to/local/model-store \
--set nim.nimCache=/model \
--set extraVolumeMounts.model-store.mountPath=/opt/cache

The host directory must be readable and writable by user ID and group ID 1000. The chart sets podSecurityContext.fsGroup to 1000 by default. Check the host directory permissions before you deploy. Host-path data is local to a node and isn’t available if the Pod moves to another node.

Alternatively, save the persistent-storage settings in embed-storage.yaml and pass -f embed-storage.yaml to Helm:

persistence:
  enabled: true
nim:
  nimCache: /model
extraVolumeMounts:
  model-store:
    mountPath: /opt/cache

For more information, refer to Parameters.

Deploying#

To deploy with persistent storage, run the following command:

helm upgrade --install \
  --namespace embedding-nim \
  nemo-embedder \
  --set persistence.enabled=true \
  --set persistence.storageClass=local-nfs \
  --set nim.nimCache=/model \
  --set extraVolumeMounts.model-store.mountPath=/opt/cache \
  nvidia-nim-nemotron-embed-2.4.0.tgz

To deploy without persistent storage, run the following command. The NIM downloads the model again each time the Pod restarts.

helm upgrade --install \
  --namespace embedding-nim \
  nemo-embedder \
  --set persistence.enabled=false \
  --set nim.nimCache=/model \
  --set extraVolumeMounts.model-store.mountPath=/opt/cache \
  nvidia-nim-nemotron-embed-2.4.0.tgz

To select a different container version, add --set image.tag=2.4.0.

After you deploy, check that the Pod is running. The initial image pull and model download can take more than 15 minutes.

kubectl get pods -n embedding-nim

The Pod eventually reaches the Running state, with output similar to the following:

NAME                              READY   STATUS    RESTARTS   AGE
nemo-embedder-nvidia-nim-nemotron-embed-0   1/1     Running   0          8m44s

Check the events for failures:

kubectl get events -n embedding-nim

Recommended Configuration for Minikube#

Use the standard storage class with the same cache mounts:

--set persistence.enabled=true \
--set persistence.storageClass=standard \
--set nim.nimCache=/model \
--set extraVolumeMounts.model-store.mountPath=/opt/cache

Running Inference#

In the previous example, the API endpoint is exposed on port 8000 through the Kubernetes service of the default type with no ingress, since authentication is not handled by the NIM itself. Port 8000 is the default service.httpPort. If you set a different service.httpPort, use that value as the second port in the following kubectl port-forward command; the container still listens on port 8000.

Use the following command to port-forward the service to your local computer to test inference.

kubectl port-forward -n embedding-nim service/nemo-embedder-nvidia-nim-nemotron-embed 8000:8000

The chart’s nim.modelName value does not select or rename the served model. Leave it empty.

These examples use nvidia/Nemotron-3.5-Embed-1B-BF16. On NVIDIA GB200, NVIDIA B300, and NVIDIA RTX PRO 6000 Blackwell Server Edition, the NIM serves nvidia/Nemotron-3.5-Embed-1B-NVFP4 by default. To confirm the model ID that your deployment serves, query /v1/models.

Then try a request:

curl -X 'POST' \
  'http://localhost:8000/v1/embeddings' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "hello world",
    "model": "nvidia/Nemotron-3.5-Embed-1B-BF16",
    "input_type": "query",
    "modality": "text"
  }'

Viewing Log Messages#

Use the following command to view the container log messages.

# Step 1: Find the actual pod name
kubectl get pods -n embedding-nim

# Step 2: Use the actual pod name from step 1
kubectl logs -f -n embedding-nim <pod-name-from-step-1>