Advanced Configuration for NeMo Evaluator

This guide provides advanced configuration options for the NeMo Evaluator microservice, such as connecting to external dependencies, customizing workflow orchestration, and enabling advanced monitoring.

---

Before You Start

Complete the Helm Chart deployment before using these advanced options. This article assumes you have a working NeMo Evaluator deployment and want to customize or extend it for production or advanced use cases.

---

NIM Proxy with External DNS

For production deployments, it is recommended to create a DNS record for NIM Proxy. Evaluator will need to resolve the NIM Proxy DNS record within the cluster to run jobs that depend on models deployed on the platform. Update the configuration for the NIM Proxy endpoints to resolve the model URLs without reconfiguring the Kubernetes cluster.

evaluator:
  external:
    nimProxy:
      # -- The DNS record for NIM Proxy.
      endpoint: "https://nim.example.com"
      # -- The internal cluster URL of the NeMo NIM Proxy microservice. This is used to remap model URLs from evaluation jobs that match external.nimProxy.endpoint to the internal URL when cluster ingress is not supported for the external address.
      endpoint_internal: "http://nemo-nim-proxy:8000"

Externalizing Dependencies

By default, the Helm chart deploys all required dependencies. For production or custom environments, you may wish to use your own Milvus, or PostgreSQL instances.

External Dependency Configuration

Milvus

evaluator:
  milvus:
    enabled: false
  external:
    milvus:
      endpoint: "<url to the pre-installed milvus>"

PostgreSQL

By default, the NeMo Entity Store Helm chart uses the Bitnami PostgreSQL chart. To use an external PostgreSQL database, set the appropriate values and see the PostgreSQL section for details.

---

Data Store Integration

To use a custom or external Data Store, specify the endpoint in your custom-values.yaml:

evaluator:
  external:
    dataStore:
      endpoint: "http://<nemo-data-store-service>.<nemo-data-store-namespace>.svc.cluster.local:8000"

---

Milvus

For Retriever and RAG evaluations with large datasets, you may want to tune Milvus storage or use your own instance. Example for default Milvus install:

evaluator:
  milvus:
    enabled: true
    serviceName: milvus
    cluster:
      enabled: false
    etcd:
      enabled: false
    pulsar:
      enabled: false
    minio:
      enabled: false
      tls:
        enabled: false
    standalone:
      persistence:
        enabled: true
        persistentVolumeClaim:
          size: 100Gi
          storageClass: standard
      extraEnv:
        - name: LOG_LEVEL
          value: error
    extraConfigFiles:
      user.yaml: |+
        etcd:
          use:
            embed: true
          data:
            dir: /var/lib/milvus/etcd
        common:
          storageType: local

Tip

For large datasets, set storage size to at least 100Gi. For production, use an external Milvus as described above.

---

Monitoring & Telemetry

NeMo Evaluator is instrumented with OpenTelemetry. To enable exporters and customize telemetry:

Basic OpenTelemetry

evaluator:
  otelExporterEnabled: true
  otelEnvVars:
    OTEL_EXPORTER_OTLP_ENDPOINT: "http://<otel-collector>:4317"
    OTEL_SERVICE_NAME: "nemo-evaluator"
    OTEL_TRACES_EXPORTER: otlp
    OTEL_METRICS_EXPORTER: otlp
    OTEL_LOGS_EXPORTER: otlp
    OTEL_PROPAGATORS: "tracecontext,baggage"
    OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=$(NAMESPACE)"
    OTEL_PYTHON_EXCLUDED_URLS: "health"
    OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: "true"

With OpenTelemetry Collector

evaluator:
  opentelemetry-collector:
    enabled: true

With Zipkin

evaluator:
  zipkin:
    enabled: true

Evaluator Monitoring & Telemetry Configuration Options

Name	Description	Value
zipkin.enabled	Deploy Zipkin for metrics.	false
opentelemetry-collector.enabled	Deploy OpenTelemetry Collector for metrics.	false
opentelemetry-collector.config	OpenTelemetry Collector configurations.	{{}}
otelExporterEnabled	Enable OpenTelemetry exporters for Evaluator.	false
otelEnvVars	Environment variables to configure OpenTelemetry for Evaluator.	{{}}
logLevel	Log level for both OTLP and console exporters.	INFO

See also: Open Telemetry Setup

---

Port Forwarding & Access

To access the Evaluator microservice locally for testing or debugging:

kubectl -n <NAMESPACE> port-forward service/myrelease-nemo-evaluator 7331:7331

After port-forwarding, your data scientists can use the local host URL to access the NeMo Evaluator microservice APIs.