Deploy NeMo Evaluator Using Docker Compose#

You can deploy the NeMo Evaluator microservice for local development or testing using Docker Compose. This method is not recommended for production but is useful for evaluation and experimentation.

Prerequisites#

Authenticate with NGC#

Before pulling container images, log in to the NVIDIA NGC container registry. The username must be the literal string $oauthtoken and the password is your NGC API key.

docker login -u '$oauthtoken' -p $NGC_API_KEY nvcr.io

Replace $NGC_API_KEY with your actual NGC API key if it is not already set as an environment variable.

Set Up Docker Compose#

Prepare Environment Variables#

Set the required environment variables for your images and user IDs:

export EVALUATOR_IMAGE=nvcr.io/nvidia/nemo-microservices/evaluator:<VERSION>
export DATA_STORE_IMAGE=nvcr.io/nvidia/nemo-microservices/datastore:<VERSION>
export USER_ID=$(id -u)
export GROUP_ID=$(id -g)
# NGC Image Pull Secret is used for image pulls (see prerequisites)

Start Evaluator#

  1. From the nemo_ms_docker_compose/ directory, run:

    docker compose -f docker_compose.yaml up evaluator -d

    This will start the following containers:

    NeMo Microservices Docker Compose Containers#

    Container

    Description

    nemo-evaluator

    Runs the Evaluator microservice.

    nemo-postgresql

    Runs PostgreSQL to manage metadata for Evaluator and other services.

    nemo_ms_docker_compose-evaluator-postgres-db-migration

    Runs database migration scripts for Evaluator.

    nemo-data-store

    Runs the NeMo Data Store microservice for managing datasets.

    nemo_ms_docker_compose-data-store-volume-init

    Initializes the volume for the Data Store.

    nemo_ms_docker_compose-otel-collector-1

    Runs OpenTelemetry Collector (optional; see below).

  2. Review the output:

    [+] Running 8/8
 ✔ Network nemo_ms_docker_compose_nemo-ms                           Created
 ✔ Network nemo_ms_docker_compose_default                           Created
 ✔ Container nemo_ms_docker_compose-otel-collector-1                Started
 ✔ Container nemo_ms_docker_compose-data-store-volume-init-1        Exited
 ✔ Container nemo-postgresql                                        Healthy
 ✔ Container nemo-data-store                                        Started
 ✔ Container nemo_ms_docker_compose-evaluator-postgres-db-migration-1  Exited
 ✔ Container nemo-evaluator                                         Started

Database Initialization#

The nemo-postgresql service uses the init_scripts/create_databases.sh script to create the required evaluation database for the Evaluator if it doesn’t exist. This is handled automatically via the compose file.

Note

The initialization script may also create additional databases (such as entity-store, ndsdb, or customizer) if they don’t already exist, but only the evaluation database is required for the Evaluator.

Configuration Options#

  • External PostgreSQL: To use an external PostgreSQL instance, set the POSTGRES_URI environment variable for the evaluator and evaluator-postgres-db-migration services, and remove services.evaluator.depends_on.nemo-postgresql from the compose file.

  • Disabling OpenTelemetry: To disable OpenTelemetry, remove services.evaluator.depends_on.otel-collector and all OTEL_ environment variables from the evaluator service.

  • Model Storage: Models are expected to be deployed externally and accessible for custom evaluations. You may map local model directories as needed.

Service Endpoints#

After starting the services with Docker Compose, the following endpoints will be available by default:

  • Evaluator API: http://localhost:7331

    • This is the main endpoint for interacting with the Evaluator microservice.

  • Nemo Data Store HuggingFace Endpoint: http://localhost:3000/v1/hf

    • The Data Store exposes a HuggingFace-compatible API at this endpoint.

    • You can set the HF_ENDPOINT environment variable to this URL if needed for integration or testing.

Stop Evaluator#

To stop Evaluator and its related services, run the following command:

docker compose -f docker_compose.yaml down

This command stops and removes the containers started for the Evaluator and its dependencies. You can restart them at any time using the up command.

Include the flag -v or --volumes to remove persistent volumes.

docker compose -f docker_compose.yaml down -v

Limitations#

  • Only custom type evaluation jobs are supported in this setup.

  • Models must be deployed externally and accessible to the Evaluator.

  • For production deployments, use Helm-based methods for scalability and security.