Deploy NeMo Evaluator Using Helm Chart#

You can deploy the NeMo Evaluator microservice using the Helm chart, which manages all required dependencies automatically. This is the recommended approach for most users.

By default, this deployment enables only the Evaluator and Data Store microservices, which are the minimal required dependencies for Evaluator. All other microservices are explicitly disabled.

Why Use the Helm Chart?

  • Dependency Management: The Helm chart ensures all required services are installed and networked correctly.

  • Simplicity: One command, one values file, minimal manual configuration.

  • Support: This is the officially supported and tested deployment method.

Prerequisites#

Dependencies

  • Kubernetes cluster with Helm installed. Check software requirements for supported versions.

  • NVIDIA GPU Cloud (NGC) API key for pulling images

    • Create an NGC API key following the instructions at Generating NGC API Keys. You need the NGC API Key to fetch Docker images required by the NeMo microservices platform.

  • (Optional) External services if not using the chart-provided dependencies:

    • NIM: NVIDIA NIM enables self-hosting of LLMs in managed environments. NIM-hosted models can serve as evaluation targets or as judge models in an LLM-as-judge setup.

    • Milvus: Milvus is an open-source vector database used for Retriever and RAG evaluations. During these evaluations, it can be used to store or query embeddings.

Kubernetes

  • Create the Kubernetes namespace where you want to install the platform if it does not already exist.

    kubectl create namespace <NAMESPACE>

  • Create NGC Image pull credentials. This includes:

    • A generic secret containing your NGC API key, referenced by existingSecret

    • An image pull secret, referenced by existingImagePullSecret

Download the Helm Chart#

Export the NGC API Key into your shell environment using the following command:

export NGC_API_KEY=<your-ngc-api-key>

Add the NeMo Microservices Helm Chart helm repo.

helm repo add nmp https://helm.ngc.nvidia.com/nvidia/nemo-microservices \
   --username='$oauthtoken' \
   --password=$NGC_API_KEY

helm repo update

Prepare Your Values File#

Create a values-evaluator-only.yaml file to enable only the Evaluator and its required dependencies. For more advanced scenarios (for example, enabling Milvus, Ingress), see the advanced example below.

tags:
  platform: false
  evaluator: true

tags:
  platform: false
  evaluator: true

evaluator:
  milvus:
    enabled: true
    standalone:
      persistence:
        enabled: true
        persistentVolumeClaim:
          size: 50Gi
          storageClass: "standard"
ingress:
  enabled: true

If you want to use external services for any dependency, set its enabled value to false and provide the appropriate external endpoint in your values file.

Install with Helm#

Run the following command to install the platform with your custom values file. Replace <NAMESPACE> and <release-name> as appropriate. An example release name can be nemo.

helm --namespace <NAMESPACE> install <release-name> \
  nmp/nemo-microservices-helm-chart \
  -f values-evaluator-only.yaml

Service Endpoints#

After deploying the services with Helm, port-forward the following services to be available:

  • Evaluator API: http://localhost:8080

    • This is the main endpoint for interacting with the Evaluator microservice.

    • Port-forward the Evaluator service. Replace <NAMESPACE> and <release-name> as appropriate.

      kubectl -n <NAMESPACE> \
  port-forward \
  service/<release-name>-evaluator \
  8080:7331

  • Nemo Data Store HuggingFace Endpoint: http://localhost:3000/v1/hf

    • The Data Store exposes a HuggingFace-compatible API at this endpoint.

    • You can set the HF_ENDPOINT environment variable to this URL if needed for integration or testing.

    • Port-forward the Evaluator service. Replace <NAMESPACE> and <release-name> as appropriate.

      kubectl -n <NAMESPACE> \
  port-forward \
  service/<release-name>-data-store \
  3000:3000

Verify the deployment:

curl http://localhost:7331/health
{"status":"healthy"}

Troubleshooting#

Secret Ownership/Label Errors#

If you encounter an error like:

Error: INSTALLATION FAILED: Unable to continue with install: Secret "nvcrimagepullsecret" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "nemo-platform"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"

  • Make sure you have set existingSecret and existingImagePullSecret in your values file as shown above.

  • Do not let Helm try to create or import secrets you created manually.

ImagePullBackOff or manifest unknown#

If pods fail to start and you encounter events like:

Failed to pull image "nvcr.io/nvidia/nemo-microservices/evaluator:${parent_helm_chart_latest_version}": Error response from daemon: manifest for nvcr.io/nvidia/nemo-microservices/evaluator:${parent_helm_chart_latest_version} not found: manifest unknown: manifest unknown

  • Check that the image tag exists in NGC.

  • Override the image tag in your values file if needed.

    evaluator:
  image:
    tag: "${parent_helm_chart_latest_version}"  # Change to a valid tag if needed

  • Ensure your image pull secret is correct and referenced properly.