Advanced Configuration Reference for NeMo Evaluator Helm Chart#

This reference guide describes advanced configuration options for the NeMo Evaluator microservice. Use these settings to connect to external dependencies, customize workflow orchestration, enable advanced monitoring, and more.

Before You Start#

Complete the Parent Helm Chart deployment before using these advanced options. This article assumes you have a working Evaluator deployment and want to customize or extend it for production or advanced use cases.

Externalizing Dependencies#

By default, the parent chart deploys all required dependencies. For production or custom environments, you may wish to use your own Argo Workflows, Milvus, or PostgreSQL instances.

External Dependency Configuration#

In your custom-values.yaml:

argoWorkflows:
  enabled: false
external:
  argoWorkflows:
    endpoint: "<url to the pre-installed argo workflows>"

milvus:
  enabled: false
external:
  milvus:
    endpoint: "<url to the pre-installed milvus>"

By default, the NeMo Entity Store Helm chart uses the Bitnami PostgreSQL chart. To use an external PostgreSQL database, set the appropriate values and see the PostgreSQL section for details.

Data Store Integration#

To use a custom or external Data Store, specify the endpoint in your custom-values.yaml:

external:
  dataStore:
    endpoint: "http://<nemo-data-store-service>.<nemo-data-store-namespace>.svc.cluster.local:8000"

Argo Workflows#

If you deploy Argo Workflows with the Evaluator chart, you can customize CRDs, service accounts, and authentication:

argoWorkflows:
  enabled: true
  serviceName: argo-workflows-server
  server:
    authModes:
      - "server"
    servicePort: 2746
    secure: true
  crds:
    install: false
argoServiceAccount:
  create: true
  name: workflow-executor

Note

Argo Workflows CRDs can only be installed once at the cluster level. For details, see the Argo Workflows documentation.

Milvus#

For Retriever and RAG evaluations with large datasets, you may want to tune Milvus storage or use your own instance. Example for default Milvus install:

milvus:
  enabled: true
  serviceName: milvus
  cluster:
    enabled: false
  etcd:
    enabled: false
  pulsar:
    enabled: false
  minio:
    enabled: false
    tls:
      enabled: false
  standalone:
    persistence:
      enabled: true
      persistentVolumeClaim:
        size: 100Gi
        storageClass: standard
    extraEnv:
      - name: LOG_LEVEL
        value: error
  extraConfigFiles:
    user.yaml: |+
      etcd:
        use:
          embed: true
        data:
          dir: /var/lib/milvus/etcd
      common:
        storageType: local

Tip

For large datasets, set storage size to at least 100Gi. For production, use an external Milvus as described above.

Monitoring & Telemetry#

NeMo Evaluator is instrumented with OpenTelemetry. To enable exporters and customize telemetry:

otelExporterEnabled: true
otelEnvVars:
  OTEL_EXPORTER_OTLP_ENDPOINT: "http://<otel-collector>:4317"
  OTEL_SERVICE_NAME: "nemo-evaluator"
  OTEL_TRACES_EXPORTER: otlp
  OTEL_METRICS_EXPORTER: otlp
  OTEL_LOGS_EXPORTER: otlp
  OTEL_PROPAGATORS: "tracecontext,baggage"
  OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=$(NAMESPACE)"
  OTEL_PYTHON_EXCLUDED_URLS: "health"
  OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: "true"

opentelemetry-collector:
  enabled: true

zipkin:
  enabled: true
Evaluator Monitoring & Telemetry Configuration Options#

Name

Description

Value

zipkin.enabled

Deploy Zipkin for metrics.

false

opentelemetry-collector.enabled

Deploy OpenTelemetry Collector for metrics.

false

opentelemetry-collector.config

OpenTelemetry Collector configurations.

{{}}

otelExporterEnabled

Enable OpenTelemetry exporters for Evaluator.

false

otelEnvVars

Environment variables to configure OpenTelemetry for Evaluator.

{{}}

logLevel

Log level for both OTLP and console exporters.

INFO

See also: Open Telemetry Setup

Port Forwarding & Access#

To access the Evaluator microservice locally for testing or debugging:

kubectl -n <NAMESPACE> port-forward service/myrelease-nemo-evaluator 7331:7331

After port-forwarding, your data scientists can use the local host URL to access the NeMo Evaluator microservice APIs.